{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":620251190,"defaultBranch":"main","name":"SharpToken","ownerLogin":"dmitry-brazhenko","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-03-28T10:17:52.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/61410067?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1715968383.0","currentOid":""},"activityList":{"items":[{"before":"b4861b23cde9d307c2286c5b373fa03ca9010601","after":null,"ref":"refs/heads/winzig/main","pushedAt":"2024-05-17T17:45:22.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"}},{"before":"c7de8c04ba38167cdbdc23a8fac4ee521c8ee97a","after":"27eef742966ec0ce75f82277fccda3e77061f4ce","ref":"refs/heads/main","pushedAt":"2024-05-17T17:45:19.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"[duplicate] Support for o200k_base and gpt-4o (omni) model (#43)\n\n* Adding o200k_base.tiktoken\r\n\r\n* Support for o200k_base and gpt-4o (omni)\r\n\r\n* Fixing typo\r\n\r\nAccidentally removed a using statement in my last update\r\n\r\n* Fixed issue with some tests\r\n\r\nStill failing in a handful of tests of the new o200k\r\n\r\n* All tests now passing (fixed typo)\r\n\r\n* architecture: x64 # Add this line\r\n\r\n* attempt\r\n\r\n* Update dotnet-build-test.yml\r\n\r\n---------\r\n\r\nCo-authored-by: Tom Winzig \r\nCo-authored-by: Tom Winzig ","shortMessageHtmlLink":"[duplicate] Support for o200k_base and gpt-4o (omni) model (#43)"}},{"before":"3b3c1a9e8a0686ad68c564fe77a0b96264404c73","after":"b4861b23cde9d307c2286c5b373fa03ca9010601","ref":"refs/heads/winzig/main","pushedAt":"2024-05-17T17:38:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"Update dotnet-build-test.yml","shortMessageHtmlLink":"Update dotnet-build-test.yml"}},{"before":"9f0e85c8209c666a691e793298e05fe8f5870832","after":"3b3c1a9e8a0686ad68c564fe77a0b96264404c73","ref":"refs/heads/winzig/main","pushedAt":"2024-05-17T16:41:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"attempt","shortMessageHtmlLink":"attempt"}},{"before":null,"after":"9f0e85c8209c666a691e793298e05fe8f5870832","ref":"refs/heads/winzig/main","pushedAt":"2024-05-17T16:32:58.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":" architecture: x64 # Add this line","shortMessageHtmlLink":" architecture: x64 # Add this line"}},{"before":"086544dcccfd36562a6eac2d5d1f603db9255925","after":"c7de8c04ba38167cdbdc23a8fac4ee521c8ee97a","ref":"refs/heads/main","pushedAt":"2024-04-08T10:37:51.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"Add pointer to Microsoft.ML.Tokenizers (#37)","shortMessageHtmlLink":"Add pointer to Microsoft.ML.Tokenizers (#37)"}},{"before":"f64efd84def29c0f6998432ee19404925aa7d59d","after":null,"ref":"refs/heads/pr-33","pushedAt":"2024-03-26T11:12:48.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"}},{"before":"e96811a19d678e8a2c1d8e8ce3704de2416e3e28","after":"086544dcccfd36562a6eac2d5d1f603db9255925","ref":"refs/heads/main","pushedAt":"2024-03-26T11:12:41.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"Pr 33 (Feature/performance: This PR introduces a high number of performance improvements) (#36)\n\n* feat(performance): add benchmark project\r\n\r\n* feat(performance): reduce public API surface\r\n\r\n* feat(performance): reduce allocations\r\n\r\n* feat(performance): use compiled regex for better performance\r\n\r\n* feat(performance): run benchmark\r\n\r\n* chore: move reorganize files\r\n\r\n* feat(performance): replace SpecialTokenPatternRegex with faster alloc free solution\r\n\r\n* feat(performance): reduce string allocations\r\n\r\n* feat(performance): run benchmark\r\n\r\n* feat(performance): add BytePairIndex class to support faster implementation in net8.0\r\n\r\n* feat(performance): cache model parameters to do params preparation only once\r\n\r\n* feat(performance): run benchmark\r\n\r\n* feat(performance): improve allowedSpecialTokens handling + bug fix\r\n\r\n* feat(performance): add support for ReadOnlySpan in net8.0\r\n\r\n* feat(performance): run benchmark\r\n\r\n* feat(performance): use compile time generated regex in net8.0\r\n\r\n* feat: run tests for netstandard2.0 over net471\r\n\r\n* feat(performance): run benchmark\r\n\r\n* feat: remove unused lastTokenLength + refactoring\r\n\r\n* feat(performance): implement fast MultiBytePairEncoder with almost zero allocations\r\n\r\n* feat(performance): run benchmark\r\n\r\n* feat(performance): reduce allocations\r\n\r\n* feat(performance): backport some optimizations to net6.0 and netstandard\r\n\r\n* feat(performance): run benchmark\r\n\r\n* chore: cosmetics\r\n\r\n* feat(refactor): extract Encoding.Encode() logic into priv EncodeCore to support broader use\r\n\r\n* feat(token-count): implement low allocation token count public method\r\n\r\n* feat(performance): run benchmark\r\n\r\n* feat(benchmark): don't make allocations in benchmark methods\r\n\r\n* feat(performance): reduce minor allocations\r\n\r\n* feat(benchmark): add benchmark for large file token count\r\n\r\n* feat(performance): run benchmark\r\n\r\n* fix: add test for allowedSpecialTokens and fix code\r\n\r\n* chore: fix naming\r\n\r\n* feat(benchmark): add another benchmark to show in README.md\r\n\r\n* feat(readme): add benchmark to README.md and add docs for TokenCount method\r\n\r\n* feat(performance): use multibyte cpu instructions for FastPartitionList.RemoveAt\r\n\r\n* feat(performance): re-add fast path - got lost in refactoring\r\n\r\n* feat(benchmark): add comparison to other tokenizer\r\n\r\n* feat(performance): improve ByteArrayEqualityComparer\r\n\r\n* feat(performance): small improvement\r\n\r\n* chore: update benchmarks in README.md\r\n\r\n* fix\r\n\r\n* improved pipleline\r\n\r\n* switch (key)\r\n\r\n* i += size - 1;\r\n\r\n* CountTokens renamed\r\n\r\n---------\r\n\r\nCo-authored-by: René Larch ","shortMessageHtmlLink":"Pr 33 (Feature/performance: This PR introduces a high number of perfo…"}},{"before":"386ca4143cdf036f30a5bd5013a0ea870cb40b93","after":"f64efd84def29c0f6998432ee19404925aa7d59d","ref":"refs/heads/pr-33","pushedAt":"2024-03-26T11:06:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"CountTokens renamed","shortMessageHtmlLink":"CountTokens renamed"}},{"before":"ab3b321bfd6a8b817e0b37ef6dcac29faebb5ef1","after":"386ca4143cdf036f30a5bd5013a0ea870cb40b93","ref":"refs/heads/pr-33","pushedAt":"2024-03-26T10:33:13.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"i += size - 1;","shortMessageHtmlLink":"i += size - 1;"}},{"before":"0c0e50537c97d9e2cec91d4e9c822ed99a98f5f7","after":"ab3b321bfd6a8b817e0b37ef6dcac29faebb5ef1","ref":"refs/heads/pr-33","pushedAt":"2024-03-26T10:31:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"switch (key)","shortMessageHtmlLink":"switch (key)"}},{"before":null,"after":"0c0e50537c97d9e2cec91d4e9c822ed99a98f5f7","ref":"refs/heads/pr-33","pushedAt":"2024-03-26T10:09:09.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"improved pipleline","shortMessageHtmlLink":"improved pipleline"}},{"before":"022fb259ae01bcabc8d8514032aa0445320de0fa","after":null,"ref":"refs/heads/fix/strongnameassemblies","pushedAt":"2024-03-25T13:24:56.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"}},{"before":"e96811a19d678e8a2c1d8e8ce3704de2416e3e28","after":null,"ref":"refs/heads/r-Larch-feature/performance","pushedAt":"2024-03-25T13:24:50.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"}},{"before":null,"after":"e96811a19d678e8a2c1d8e8ce3704de2416e3e28","ref":"refs/heads/r-Larch-feature/performance","pushedAt":"2024-03-25T13:23:57.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"Pipeline fix (#35)","shortMessageHtmlLink":"Pipeline fix (#35)"}},{"before":"afa99f376661d7628d9a6e368d051903385f073b","after":null,"ref":"refs/heads/build_pipeline_fix","pushedAt":"2024-03-25T12:47:13.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"}},{"before":"5b48c72b6cdec7f489d72cb5c500643cf63f3ec0","after":"e96811a19d678e8a2c1d8e8ce3704de2416e3e28","ref":"refs/heads/main","pushedAt":"2024-03-25T12:47:10.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"Pipeline fix (#35)","shortMessageHtmlLink":"Pipeline fix (#35)"}},{"before":null,"after":"afa99f376661d7628d9a6e368d051903385f073b","ref":"refs/heads/build_pipeline_fix","pushedAt":"2024-03-25T12:46:36.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}},{"before":"04d9519ccf1c1eee38a560668c9f7fbd7652a66b","after":null,"ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T12:43:58.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"}},{"before":"1cae79cf643d56ed0ed7c7d7e12e021807ac8a20","after":"5b48c72b6cdec7f489d72cb5c500643cf63f3ec0","ref":"refs/heads/main","pushedAt":"2024-03-25T12:43:54.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"Pipelines update (#34)\n\n* workflows\r\n\r\n* fix\r\n\r\n* fix\r\n\r\n* fix\r\n\r\n* fix\r\n\r\n* fix\r\n\r\n* fix\r\n\r\n* fix 1\r\n\r\n* fix 2\r\n\r\n* fix\r\n\r\n* fix4\r\n\r\n* fix\r\n\r\n* dotnet-version fix\r\n\r\n* needs: call-reusable-build-test\r\n\r\n* actions/checkout@v4\r\n\r\n* actions/setup-dotnet@v4","shortMessageHtmlLink":"Pipelines update (#34)"}},{"before":"6cf7b51c0a7084a6d62e9a36c55aa7c92a8761a4","after":"04d9519ccf1c1eee38a560668c9f7fbd7652a66b","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T12:09:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"actions/setup-dotnet@v4","shortMessageHtmlLink":"actions/setup-dotnet@v4"}},{"before":"ec545b7c44b60efd567250b76c9a86418159c510","after":"6cf7b51c0a7084a6d62e9a36c55aa7c92a8761a4","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T12:06:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"actions/checkout@v4","shortMessageHtmlLink":"actions/checkout@v4"}},{"before":"e97481c8105bbf8d6ae9462c6c648660635078d6","after":"ec545b7c44b60efd567250b76c9a86418159c510","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T12:03:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"needs: call-reusable-build-test","shortMessageHtmlLink":"needs: call-reusable-build-test"}},{"before":"0bb52de56daa768866dc8759735d05a34956cd4c","after":"e97481c8105bbf8d6ae9462c6c648660635078d6","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T11:59:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"dotnet-version fix","shortMessageHtmlLink":"dotnet-version fix"}},{"before":"eab21bd12fa4bc2240b8acdf5882edc19aae41db","after":"0bb52de56daa768866dc8759735d05a34956cd4c","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T11:47:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}},{"before":"a6439b9ebbf64e59d4f5aaa73cfabaa3b58b5487","after":"eab21bd12fa4bc2240b8acdf5882edc19aae41db","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T11:36:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"fix4","shortMessageHtmlLink":"fix4"}},{"before":"5d18f68b252b222c34ff989b91da4bcb48eca3ed","after":"a6439b9ebbf64e59d4f5aaa73cfabaa3b58b5487","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T11:34:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}},{"before":"4154ef6616cf0e1dd90278dd1814d1797e9a0ce0","after":"5d18f68b252b222c34ff989b91da4bcb48eca3ed","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T11:30:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"fix 2","shortMessageHtmlLink":"fix 2"}},{"before":"7b1bbb7b89e3e96a83a12bf121e83359c997a9b2","after":"4154ef6616cf0e1dd90278dd1814d1797e9a0ce0","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T11:29:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"fix 1","shortMessageHtmlLink":"fix 1"}},{"before":"d6c2f2c3f6beefbc96802f26c47f46ea2398bb19","after":"7b1bbb7b89e3e96a83a12bf121e83359c997a9b2","ref":"refs/heads/build_test_pipeline","pushedAt":"2024-03-25T11:28:13.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dmitry-brazhenko","name":"Dmitry Brazhenko","path":"/dmitry-brazhenko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/61410067?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAETSpUgQA","startCursor":null,"endCursor":null}},"title":"Activity · dmitry-brazhenko/SharpToken"}