{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":763697768,"defaultBranch":"main","name":"veScale","ownerLogin":"volcengine","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2024-02-26T19:01:27.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/67365215?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1724625802.0","currentOid":""},"activityList":{"items":[{"before":"93fe1d507038e16926e4283a4645a08dfab2ad21","after":null,"ref":"refs/heads/open_source_081324","pushedAt":"2024-08-25T22:43:22.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"pengyanghua","name":"yhpeng","path":"/pengyanghua","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13825158?s=80&v=4"}},{"before":"e439aa91919207c54e436659990a91aa6251b7e0","after":"b4b1686fecd2805a98a89cd7813ac2fa4957837f","ref":"refs/heads/main","pushedAt":"2024-08-25T22:43:20.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"pengyanghua","name":"yhpeng","path":"/pengyanghua","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13825158?s=80&v=4"},"commit":{"message":"feat: add nccl stream fetch api and add dependency version limit (#48)\n\n1. add nccl stream fetch api in pytorch patches\r\n2. add dependency version limit about numpy and pytest in torch_patch\r\nand vescale requirements","shortMessageHtmlLink":"feat: add nccl stream fetch api and add dependency version limit (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2475335516\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/48\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/48/hovercard\" href=\"https://github.com/volcengine/veScale/pull/48\">#48</a>)"}},{"before":null,"after":"93fe1d507038e16926e4283a4645a08dfab2ad21","ref":"refs/heads/open_source_081324","pushedAt":"2024-08-20T10:50:24.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"vocaltract","name":"noob_wcy","path":"/vocaltract","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55313095?s=80&v=4"},"commit":{"message":"feat: add nccl stream fetch api; chore: add dependency version limit both in torch_patch and vescale requirements","shortMessageHtmlLink":"feat: add nccl stream fetch api; chore: add dependency version limit …"}},{"before":"70db7e72e98ce9818ea7b515656d35449450201a","after":"e439aa91919207c54e436659990a91aa6251b7e0","ref":"refs/heads/main","pushedAt":"2024-08-10T04:24:22.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"[emulator] feat: veScale correctness emulator (#45)\n\nThis pull request contains **veScale Correctness Emulator** that\r\nemulates the results from multiple devices execution on a single device.\r\n\r\n## Why veScale Correctness Emulator?\r\n- Modern Frameworks promise **Single-Device Abstraction** for **nD\r\nParallelism**. But it is still missing a critical component that can\r\nverify the ***correctness*** of **Single-Device Abstraction of nD\r\nParallelism**. For example, there are differences between the loss curve\r\nof single device training and loss curves of 3D parallelism training.\r\n- How do we know the difference is *correct*? To what extent is it\r\n*correct*?\r\n    - \"Correct\" differences come from nD Parallelism\r\n        - Communication difference (e.g., ring allreduce)\r\n        - Compute difference (e.g., matmul)\r\n        - Hardware difference (e.g. FP16)\r\n    - \"Incorrect\" differences come from bugs in\r\n        - User configuration\r\n        - User model code\r\n        - System implementation code\r\n        - Data loader\r\n        - Model checkpoint\r\n        - Random seed and offset\r\n\r\n## What is veScale Correctness Emulator?\r\n\r\n- **veScale Correctness Emulator** verifies nD prarllelism correctness\r\nby emulating nD parallel training on a single device,\r\n- **veScale Correctness Emulator** isolates correctness at different\r\nlayers and seperates differences come from nD parallelism with\r\ndifferences come from bugs.\r\n- **veScale Correctness Emulator** achieves bitwise correctness in three\r\nlevels: NCCL collectives, mesh collectives, and DTensor.\r\n\r\n### NCCL Emulation\r\nWe are using the NCCL version 2.19.3 code as a reference for our\r\nemulation implementation. The code can be found at\r\n[NVIDIA/nccl](https://github.com/NVIDIA/nccl/tree/v2.19.3-1).\r\n\r\n**veScale Correctness Emulator** can perfectly emulate NCCL collective\r\nAPIs' results. This is achieved by implementing the same NCCL collective\r\nalgorithms and modeling NCCL's computation order via calculating the\r\ncorrect chunk size.\r\n\r\n### Collective APIs Emulation\r\nThese are standalone collective APIs which emulate the results from\r\ncollective APIs of NCCL on a single device.\r\nSupported APIs:\r\n- `all_reduce`\r\n- `all_gather`\r\n- `reduce_scatter`\r\n- `all_to_all`\r\n\r\n### Mesh Collective APIs Emulation\r\nThese are standalone mesh collective APIs which emulate the results from\r\nmesh collective APIs of PyTorch on a single device.\r\nSupported APIs:\r\n- `mesh_all_reduce`\r\n- `mesh_all_gather`\r\n- `mesh_reduce_scatter`\r\n- `mesh_all_to_all`\r\n- `mesh_broadcast`\r\n- `mesh_scatter`\r\n\r\n### DTensor Redistribution Function Emulation\r\nThese are standalone DTensor redistribution functions which emulate the\r\nresults from DTensor redistribution functions of PyTorch on a single\r\ndevice.\r\n- `R2R`\r\n- `R2S`\r\n- `S2R`\r\n- `P2R`\r\n\r\nComming soon: A full list of emulator DTensor redistribution functions\r\nwill be added to support nD parallelisms including DP, TP, SP, PP, EP,\r\nand OP.\r\n\r\n## How does veScale Correctness Emulator work?\r\n**veScale Correctness Emulator** achieves bitwise correctness in\r\nemulating NCCL collectives APIs results. This is done by implementing\r\nthe same NCCL collective algorithms and modeling NCCL's algorithm and\r\nprotocol selection function and chunk size calculation process to ensure\r\nthe same computation order as NCCL.\r\n\r\nBased on the emulation functions for NCCL collectives, **veScale\r\nCorrectness Emulator** implements a global-view emulator `ProcessGroup`\r\nand `DeviceMesh` that contain all the process groups in the enviroment,\r\nwhile PyTorch's `ProcessGroup` and `DeviceMesh` only view process groups\r\nrelated to the current ranks.\r\n\r\nAided by the global-view emulator `ProcessGroup` and `DeviceMesh`,\r\n**veScale Correctness Emulator** can emulate the results of collective\r\nAPIs, mesh collective APIs, and DTensor redistribution functions on a\r\nsingle device.","shortMessageHtmlLink":"[emulator] feat: veScale correctness emulator (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2458683724\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/45\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/45/hovercard\" href=\"https://github.com/volcengine/veScale/pull/45\">#45</a>)"}},{"before":"9dbdd32cd9c5a6b0823ad7cde36542c7b7a95095","after":null,"ref":"refs/heads/open_source_0725_patch","pushedAt":"2024-07-31T03:57:30.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"}},{"before":"aa95bb7f1718458b2e0c2b16b56f3bff7abbf1cc","after":"70db7e72e98ce9818ea7b515656d35449450201a","ref":"refs/heads/main","pushedAt":"2024-07-31T03:57:28.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"Open source 0725 patch (#42)","shortMessageHtmlLink":"Open source 0725 patch (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2438771021\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/42\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/42/hovercard\" href=\"https://github.com/volcengine/veScale/pull/42\">#42</a>)"}},{"before":"056066ebf72d67b49d67abddc5cb991cd1a02d4b","after":"9dbdd32cd9c5a6b0823ad7cde36542c7b7a95095","ref":"refs/heads/open_source_0725_patch","pushedAt":"2024-07-31T00:27:03.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"Merge branch 'open_source_0725_patch' of github.com:volcengine/veScale into open_source_0725_patch","shortMessageHtmlLink":"Merge branch 'open_source_0725_patch' of github.com:volcengine/veScal…"}},{"before":"e34a280e39dedb1c093938e2e61ca81465246ce6","after":null,"ref":"refs/heads/revert-41-open_source_0725","pushedAt":"2024-07-30T22:52:26.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"leonardo0lyj","name":"Youjie Li","path":"/leonardo0lyj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16678974?s=80&v=4"}},{"before":null,"after":"e34a280e39dedb1c093938e2e61ca81465246ce6","ref":"refs/heads/revert-41-open_source_0725","pushedAt":"2024-07-30T22:33:23.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"leonardo0lyj","name":"Youjie Li","path":"/leonardo0lyj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16678974?s=80&v=4"}},{"before":"0abd3732ad2b64eb3a36d871a70e3b09abe9e63b","after":"056066ebf72d67b49d67abddc5cb991cd1a02d4b","ref":"refs/heads/open_source_0725_patch","pushedAt":"2024-07-30T22:23:58.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"Merge branch 'main' into open_source_0725_patch","shortMessageHtmlLink":"Merge branch 'main' into open_source_0725_patch"}},{"before":null,"after":"0abd3732ad2b64eb3a36d871a70e3b09abe9e63b","ref":"refs/heads/open_source_0725_patch","pushedAt":"2024-07-30T22:18:50.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"patch","shortMessageHtmlLink":"patch"}},{"before":"3964e83177fe9be3ad371fdad0125bd046a0ceb2","after":null,"ref":"refs/heads/open_source_0725","pushedAt":"2024-07-30T21:05:36.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"}},{"before":"c4afc72aea866239fe688079a8607a5f95874fec","after":"aa95bb7f1718458b2e0c2b16b56f3bff7abbf1cc","ref":"refs/heads/main","pushedAt":"2024-07-30T21:05:34.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"PP API and nD Distributed Timeline Profiling (#41)","shortMessageHtmlLink":"PP API and nD Distributed Timeline Profiling (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2436782929\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/41\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/41/hovercard\" href=\"https://github.com/volcengine/veScale/pull/41\">#41</a>)"}},{"before":"e207fea5150fbfa0a842ff882d2561c1ac25630d","after":"3964e83177fe9be3ad371fdad0125bd046a0ceb2","ref":"refs/heads/open_source_0725","pushedAt":"2024-07-30T18:22:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"update","shortMessageHtmlLink":"update"}},{"before":"a7042cbb8f99b69b1fd9d89e7a40926c3d13ee08","after":"e207fea5150fbfa0a842ff882d2561c1ac25630d","ref":"refs/heads/open_source_0725","pushedAt":"2024-07-30T03:12:34.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"added matplotlib","shortMessageHtmlLink":"added matplotlib"}},{"before":null,"after":"a7042cbb8f99b69b1fd9d89e7a40926c3d13ee08","ref":"refs/heads/open_source_0725","pushedAt":"2024-07-30T03:01:41.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"MackZackA","name":null,"path":"/MackZackA","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14083881?s=80&v=4"},"commit":{"message":"fix: updpate code","shortMessageHtmlLink":"fix: updpate code"}},{"before":"f645c306500723aa74c01ea8794c1a6135241a4b","after":null,"ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T07:12:57.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"}},{"before":"55c7f8a24206c3768009ed147e05723e2d69581f","after":"c4afc72aea866239fe688079a8607a5f95874fec","ref":"refs/heads/main","pushedAt":"2024-05-31T07:12:56.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"},"commit":{"message":"[checkpoint] feat: open source fast checkpoint system (#38)\n\n## Summary\r\n\r\nWe improved `vescale.checkpoint` with the following new features for\r\nfast checkpointing (where front three features are built-in techniques\r\nwithout necessitating manual activation):\r\n\r\n- **Saving Plan Caching**: During training, the program may save model\r\nand optimizer checkpoints every n steps. Once a saving plan is created,\r\nit remains unchanged as long as the model does. We implemented plan\r\ncaching to avoid regenerating the plan when checkpointing a model or\r\noptimizer multiple times, reducing unnecessary compute and communication\r\ncosts. As of 05/30/2024, PyTorch DCP does not support plan caching.\r\n\r\n- **Saving Plan Load-Balancing**: In data parallel training, models are\r\nreplicated across GPUs with different data parallel ranks but the same\r\npipeline and tensor parallel ranks. Existing PyTorch DCP (as of\r\n05/30/2024) deduplicates replicated tensors using a simple algorithm,\r\ncausing GPUs with data parallel rank 0 to save the entire model, leading\r\nto load imbalance. We implemented a load-balancing algorithm to address\r\nthis issue when deduplicating model tensors.\r\n\r\n- **D2H Tensor Copying via Pinned Memory**: When copying tensors from\r\nGPU to host memory, `vescale.checkpoint` uses pinned host memory,\r\nreducing memory allocation costs each time a checkpoint is saved. As of\r\n05/30/2024, PyTorch DCP does not support pinned memory.\r\n\r\n- **Checkpoint Broadcasting**: In data parallel training, models are\r\nreplicated across GPUs with different data parallel ranks but the same\r\npipeline and tensor parallel ranks. If `broadcast_checkpoint` is\r\nenabled, `vescale.checkpoint.load` lets GPUs with data parallel rank 0\r\nto load the model and broadcast it to other GPUs with higher data\r\nparallel ranks. If GPUs are connected with NCCL and I/O bandwidth is\r\nfully utilized, broadcasting model tensors speeds up checkpoint loading\r\ncompared to all GPUs loading models from persistent storage. E.g.:\r\n\r\n    ```python\r\n    # prepare checkpoint state for the model and optimizer\r\ncheckpoint_state = { \"model\": distributed_model, \"optimizer\":\r\ndistributed_optimizer }\r\n    # load the checkpoint\r\nvescale.checkpoint.load(\"/user/vescale/gpt/\", checkpoint_state,\r\nbroadcast_checkpoint=True)\r\n    ```\r\n\r\n- **Asynchronous Checkpointing**: When `vescale.checkpoint.save` is\r\ncalled, it first generates a saving plan and then synchronously copies\r\ntensors from GPU to host memory. If `async_checkpoint` is enabled, the\r\ntraining program can continue after the D2H copying, while\r\n`vescale.checkpoint.save` continues to serialize tensors and dump the\r\ncheckpoint to persistent storage asynchronously without blocking\r\ntraining. As of 05/30/2024, PyTorch DCP does not support asynchronous\r\ncheckpointing. E.g.:\r\n\r\n    ```python\r\n    # prepare checkpoint state for the model and optimizer\r\ncheckpoint_state = { \"model\": distributed_model, \"optimizer\":\r\ndistributed_optimizer }\r\n    # save the checkpoint asynchronuously\r\nvescale.checkpoint.save(\"/user/vescale/gpt/\", checkpoint_state,\r\nasync_checkpoint=True)\r\n    ```\r\n\r\n## Acknowledgement\r\n\r\nWe sincerely appreciate all contributors including but not limited to\r\n@shanesyy-1992 @raywan-110 @lazychao @AHEADer @MingjiHan99","shortMessageHtmlLink":"[checkpoint] feat: open source fast checkpoint system (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2326972607\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/38\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/38/hovercard\" href=\"https://github.com/volcengine/veScale/pull/38\">#38</a>)"}},{"before":null,"after":"f645c306500723aa74c01ea8794c1a6135241a4b","ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T05:00:52.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"},"commit":{"message":"open source fast checkpoint system","shortMessageHtmlLink":"open source fast checkpoint system"}},{"before":"28ddefa528cc9e45cf051adfb5b7e48598b45609","after":null,"ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T04:58:03.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"}},{"before":null,"after":"28ddefa528cc9e45cf051adfb5b7e48598b45609","ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T04:47:52.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"}},{"before":"f044f7214934c3df91193f7663ac60cdf1b017b5","after":null,"ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T04:45:59.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"}},{"before":null,"after":"f044f7214934c3df91193f7663ac60cdf1b017b5","ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T04:34:10.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"}},{"before":"a32b3347076eebc6263a2c27497a55bd064f090d","after":null,"ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T03:48:44.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"}},{"before":null,"after":"a32b3347076eebc6263a2c27497a55bd064f090d","ref":"refs/heads/opensource_053024","pushedAt":"2024-05-31T02:46:35.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"MingjiHan99","name":"MingjiHan","path":"/MingjiHan99","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9149201?s=80&v=4"}},{"before":"2a072bfe2a4697f934325c0ad415a420691146f6","after":"55c7f8a24206c3768009ed147e05723e2d69581f","ref":"refs/heads/main","pushedAt":"2024-05-24T03:43:38.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"leonardo0lyj","name":"Youjie Li","path":"/leonardo0lyj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16678974?s=80&v=4"},"commit":{"message":"[Example] Add comments to example codes (#36)\n\nIn this PR, we add comments explaining VeScale APIs in the nanoGPT\r\nexample.","shortMessageHtmlLink":"[Example] Add comments to example codes (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2311825598\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/36\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/36/hovercard\" href=\"https://github.com/volcengine/veScale/pull/36\">#36</a>)"}},{"before":"9047a730c08c2e467d2bfd624891b19b4fc28513","after":"2a072bfe2a4697f934325c0ad415a420691146f6","ref":"refs/heads/main","pushedAt":"2024-05-21T17:51:42.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"leonardo0lyj","name":"Youjie Li","path":"/leonardo0lyj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16678974?s=80&v=4"},"commit":{"message":"[DTensor&DModule&DDP&Examples] feature updates and new examples (#35)\n\nIn this PR, we add two examples and update some features in DTensor,\r\nDModule, and DDP.\r\n\r\n## Examples\r\n\r\n1. 4D finetuning the llama2_3b model.\r\n2. 4D pretraining a mixtral MOE-based model\r\n\r\n## DTensor\r\n\r\n1. Update op strategies on `Partial`ed and `InterleavedShard`ed\r\ndtensors.\r\n2. Add all-to-all communications.\r\n\r\n## DModule\r\n\r\n1. Support factory methods for nested submodules\r\n\r\n## DDP\r\n\r\n1. Unblock gradient allreduce for sparse modules in DDP","shortMessageHtmlLink":"[DTensor&amp;DModule&amp;DDP&amp;Examples] feature updates and new examples (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2304999975\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/35\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/35/hovercard\" href=\"https://github.com/volcengine/veScale/pull/35\">#35</a>)"}},{"before":"aa390759e132cdaebfbc23585c53d4e21cd579cd","after":null,"ref":"refs/heads/youjie/readme","pushedAt":"2024-05-12T01:31:59.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"pengyanghua","name":"yhpeng","path":"/pengyanghua","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13825158?s=80&v=4"}},{"before":"dd44ba5effb3c3ce3ee26b9515eea982567e351e","after":"9047a730c08c2e467d2bfd624891b19b4fc28513","ref":"refs/heads/main","pushedAt":"2024-05-12T01:31:59.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"pengyanghua","name":"yhpeng","path":"/pengyanghua","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13825158?s=80&v=4"},"commit":{"message":"[docs] fix: update README (#34)","shortMessageHtmlLink":"[docs] fix: update README (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2291097137\" data-permission-text=\"Title is private\" data-url=\"https://github.com/volcengine/veScale/issues/34\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/volcengine/veScale/pull/34/hovercard\" href=\"https://github.com/volcengine/veScale/pull/34\">#34</a>)"}},{"before":null,"after":"aa390759e132cdaebfbc23585c53d4e21cd579cd","ref":"refs/heads/youjie/readme","pushedAt":"2024-05-11T23:27:02.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"leonardo0lyj","name":"Youjie Li","path":"/leonardo0lyj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16678974?s=80&v=4"},"commit":{"message":"update","shortMessageHtmlLink":"update"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEo7a-GAA","startCursor":null,"endCursor":null}},"title":"Activity · volcengine/veScale"}