Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

Public code samples and resources for the Thales CipherTrust Application Protection products of the CipherTrust Data Security Platform

Tokenizers and Machine Learning Models for biological sequence data

Contrastive-LSH Embedding and Tokenization Technique for Multivariate Time Series Classification

A simple way for tokenization of the Real World Assets

💫 Industrial-strength Natural Language Processing (NLP) in Python

OmniTokenizer: one model and one weight for image-video joint tokenization.

Applying RWA features to the stocks. I am creating a fully reserved tokenized stock(rwa), which is collateralised by the real stock.

Rule engine used by the CMTAT token framework to implement transfer restriction.

TextSummarizer is an AI-powered tool designed to generate concise summaries from longer pieces of text. Leveraging the power of Natural Language Processing (NLP) and machine learning techniques, this tool helps in extracting the most relevant information, making it easier to understand and digest large amounts of data quickly.

TokenEx iOS SDK

The Implementation of The Ledger of Things Node. Layer 1 decentralized blockchain platform for the tokenization of objects. Proof of Scan (PoW ASIC resistant + PoS) is a revolutionary protocol preventing assets form copying. Useful smart-contracts and dApps.

Rosette API Client Library for Java

A web3 application for making fractional nfts

Practice using and preparing datasets for training from Hugging Face

A modular resource tokenization service.

Basis Theory Developer Documentation

VGS Collect iOS SDK

📐 GPT token estimation and context size utilities without a full tokenizer

Sudachi in Rust 🦀 and new generation of SudachiPy