{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":669255584,"defaultBranch":"main","name":"openai-tiktoken","ownerLogin":"whatif-dev","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-07-21T18:24:20.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/139566763?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1689963871.0","currentOid":""},"activityList":{"items":[{"before":"39f29cecdb6fc38d9a3434e5dd15e4de58cf3c80","after":"1b9faf2779855124f05174adf1383e53689ed94b","ref":"refs/heads/main","pushedAt":"2024-03-08T11:55:31.000Z","pushType":"push","commitsCount":14,"pusher":{"login":"whatif-dev","name":"WhatIfDev","path":"/whatif-dev","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/139566763?s=80&v=4"},"commit":{"message":"Simplify byte_pair_merge (#255)\n\nBased on suggestion in https://github.com/openai/tiktoken/pull/239\r\n(specifically 8f5dd7d)\r\n\r\nLike that commit, this:\r\n- Does the init in a single loop and saves a loop if there are no merges\r\n- Simplifies get_rank and no longer uses it in init (so you don't need\r\nmultiple skip values)\r\n\r\nUnlike that commit:\r\n- We drop optimisations enabled by ignoring single tokens. These didn't\r\nshow any benefit on benchmarks for me (this makes sense given typical\r\npiece sizes, but let me know if that's unexpected!). Given this, I opted\r\nfor the simpler version.\r\n- I preserve some of the comments from the original that I think are\r\nstill useful\r\n\r\nCo-authored-by: @paplorinc\r\n\r\n---------\r\n\r\nCo-authored-by: Lőrinc Pap <1841944+paplorinc@users.noreply.github.com>","shortMessageHtmlLink":"Simplify byte_pair_merge (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2127953732\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/tiktoken/issues/255\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/tiktoken/pull/255/hovercard\" href=\"https://github.com/openai/tiktoken/pull/255\">openai#255</a>)"}},{"before":"5d970c1100d3210b42497203d6b5c1e30cfda6cb","after":"39f29cecdb6fc38d9a3434e5dd15e4de58cf3c80","ref":"refs/heads/main","pushedAt":"2023-10-01T07:26:54.000Z","pushType":"push","commitsCount":4,"pusher":{"login":"whatif-dev","name":"WhatIfDev","path":"/whatif-dev","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/139566763?s=80&v=4"},"commit":{"message":"Sync codebase","shortMessageHtmlLink":"Sync codebase"}},{"before":null,"after":"e22c440e1b9c4dbae70798298c07f51744e39887","ref":"refs/heads/sourcery/main","pushedAt":"2023-07-21T18:24:31.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"sourcery-ai[bot]","name":null,"path":"/apps/sourcery-ai","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/48477?s=80&v=4"},"commit":{"message":"'Refactored by Sourcery'","shortMessageHtmlLink":"'Refactored by Sourcery'"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEEBpCWwA","startCursor":null,"endCursor":null}},"title":"Activity · whatif-dev/openai-tiktoken"}