ReadySet is a SQL caching engine designed to help developers enhance the performance and scalability of their existing database-backed applications.
ReadySet is a lightweight SQL caching engine that sits between your application and Postgres or MySQL database and turns even the most complex SQL reads into lightning-fast lookups. Unlike other caching solutions, ReadySet keeps the cache up-to-date automatically and requires no changes to your application code.
Based on years of dataflow research at MIT1, ReadySet stores the results of queries in-memory and automatically keeps these results up-to-date as the underlying database changes. ReadySet can do this automatically because it listens to your database's replication stream.
This means:
- No extra code to keep your cache and database in sync
- No extra code to evict stale records
- No TTLs to set - your cache is as up-to-date as your replication lag
ReadySet is wire-compatible with MySQL and Postgres.
These instructions show you how to run ReadySet against a new Postgres or MySQL database using Docker. You can follow along in the docs for the ReadySet Quickstart.
- Install and start Docker for your OS.
- Install the
psql
shell or themysql
shell.
Create a Docker container and start the database inside it.
For Postgres:
docker run -d \
--name=postgres \
--publish=5432:5432 \
-e POSTGRES_PASSWORD=readyset \
-e POSTGRES_DB=testdb \
postgres:14 \
-c wal_level=logical
For MySQL:
docker run -d \
--name=mysql \
--publish=3306:3306 \
-e MYSQL_ROOT_PASSWORD=readyset \
-e MYSQL_DATABASE=testdb \
mysql
Download a sample data file and load it into the database via the SQL shell.
For Postgres:
curl -O https://raw.githubusercontent.com/readysettech/docs/main/docs/assets/quickstart-data-postgres.sql
PGPASSWORD=readyset psql \
--host=127.0.0.1 \
--port=5432 \
--username=postgres \
--dbname=testdb
\i quickstart-data-postgres.sql
\q
For MySQL:
curl -O https://raw.githubusercontent.com/readysettech/docs/main/docs/assets/quickstart-data-mysql.sql
mysql \
--host=127.0.0.1 \
--port=3306 \
--user=root \
--password=readyset \
--database=testdb
source quickstart-data-mysql.sql;
\q
Create a Docker container and start ReadySet inside it, connecting ReadySet to the database via the connection string in --upstream-db-url
:
For Postgres:
docker run -d \
--name=readyset \
--publish=5433:5433 \
--platform=linux/amd64 \
--volume='readyset:/state' \
--pull=always \
-e DEPLOYMENT_ENV=quickstart_github \
public.ecr.aws/readyset/readyset:beta-2023-07-26 \
--standalone \
--deployment='github-postgres' \
--database-type=postgresql \
--upstream-db-url=postgresql:https://postgres:[email protected]:5432/testdb \
--address=0.0.0.0:5433 \
--username='postgres' \
--password='readyset' \
--db-dir='/state'
For MySQL:
docker run -d \
--name=readyset \
--publish=3307:3307 \
--platform=linux/amd64 \
--volume='readyset:/state' \
--pull=always \
-e DEPLOYMENT_ENV=quickstart_github \
public.ecr.aws/readyset/readyset:beta-2023-07-26 \
--standalone \
--deployment='github-mysql' \
--database-type=mysql \
--upstream-db-url=mysql:https://root:[email protected]:3306/testdb \
--address=0.0.0.0:3307 \
--username='root' \
--password='readyset' \
--db-dir='/state'
-
Connect an existing database
The tutorial above runs ReadySet against a sample database. To see the power of ReadySet against your own database, swap the
upstream-db-url
argument to point at an existing database for testing. -
Connect your application
Once you have a ReadySet instance up and running, the next step is to connect your application by swapping out your database connection string to point to ReadySet instead. The specifics of how to do this vary by database client library, ORM, and programming language. See Connect an App for examples.
Note: By default, ReadySet will proxy all queries to the database, so changing your app to connect to ReadySet should not impact performance. You will explicitly tell ReadySet which queries to cache.
-
Cache queries
Once ReadySet is proxying queries, connect a database SQL shell to ReadySet and use the custom
SHOW PROXIED QUERIES
SQL command to view the queries that ReadySet has proxied to your upstream database and identify which queries are supported by ReadySet. Then use the customCREATE CACHE
SQL command to cache supported queries.Note: To successfully cache the results of a query, ReadySet must support the SQL features and syntax in the query. For more details, see SQL Support.
-
Tear down
When you are done testing, stop and remove the Docker resources:
docker rm -f readyset postgres mysql \ && docker volume rm readyset
-
Additional deployment options In addition to Docker, you can deploy ReadySet using binaries. For Kubernetes environments, check out how to deploy ReadySet to Kubernetes.
Q: How does ReadySet work under the hood?
A: The heart of ReadySet is a query engine based on partially-stateful, streaming data flow. To learn more about how it works, see our docs.
Q: How does ReadySet keep cached state up to date?
A: ReadySet receives updates about data changes from your backing database via binlog replication and uses those updates to automatically update its internal state.
Q: Do I have to send all of my database traffic to ReadySet?
A: You can if you want to, but it’s not required. You can manually route a subset of your traffic to ReadySet (as you would with a traditional database read replica), or you can send all of it to ReadySet. It's important to note that not all of the queries sent to ReadySet need to be cached– you have fine-grained control over what is cached vs. what is proxied.
Q: Does ReadySet automatically determine which queries will be cached?
A: No - by default, ReadySet proxies queries to your upstream database so you can profile your application. Only after you run CREATE CACHE
will ReadySet begin caching a query.
For more information, check out our full docs site here.
For questions or support, join us on the ReadySet Community Discord to chat with our team.
This section is for developers who want to build ReadySet from source as they work on the ReadySet codebase.
-
Install ReadySet dependencies.
macOS with homebrew:
brew install lz4 [email protected]
Add the following to your cargo config to make it discover
lz4
:[env] LIBRARY_PATH = "/opt/homebrew/lib"
Ubuntu:
sudo apt update && sudo apt install -y build-essential libssl-dev pkg-config llvm clang liblz4-dev sudo apt-get -y install cmake
Arch:
sudo pacman -S base-devel clang lz4
CentOS/Amazon Linux:
sudo yum -y update sudo yum -y groupinstall "Development Tools" sudo yum -y install clang lz4-devel openssl-devel
Nix:
nix-shell
-
Install Rust via rustup.rs.
ReadySet is written entirely in Rust.
-
Install Docker via Get Docker and Docker Compose via Install Docker Compose.
ReadySet runs alongside a backing Postgres or MySQL database and, when run in distributed fashion, uses Consul for leader election, failure detection, and internal cluster state management. You'll use Docker Compose to create and manage these resources for local development.
-
Clone the repo using
git
and navigate into it:git clone https://github.com/readysettech/readyset.git cd readyset
-
Start a backing database:
docker-compose up -d
This starts both Postgres and MySQL in containers, although you will run ReadySet against only one at a time. If you don't want to run both databases, edit
docker-compose.yml
and comment out themysql
orpostgres
fields. -
Compile and run ReadySet, replacing
<deployment name>
with a unique identifier for the deployment.Run against Postgres:
cargo run --bin readyset --release -- --standalone --database-type=postgresql --upstream-db-url=postgresql:https://postgres:[email protected]:5432/testdb --username=postgres --password=readyset --address=0.0.0.0:5433 --deployment=<deployment name> --prometheus-metrics --query-log --query-log-ad-hoc
Run against MySQL:
cargo run --bin readyset --release -- --standalone --database-type=mysql --upstream-db-url=mysql:https://root:[email protected]:3306/testdb --username=root --password=readyset --address=0.0.0.0:3307 --deployment=<deployment name> --prometheus-metrics --query-log --query-log-ad-hoc
This runs the ReadySet Server and Adapter as a single process, a simple, standard way to run ReadySet that is also the easiest approach when developing. For production deployments, however, it is possible to run the Server and Adapter as separate processes. See the scale out deployment pattern in the docs.
For details about ReadySet options, see the CLI docs (coming soon).
-
With ReadySet up and running, you can now connect the Postgres or MySQL shell:
Postgres:
PGPASSWORD=readyset psql --host=127.0.0.1 \ --port=5433 \ --username=postgres \ --dbname=testdb
MySQL:
mysql \ --host=127.0.0.1 \ --port=3307 \ --user=root \ --password=readyset \ --database=testdb
To run tests for the project, run the following command:
cargo test --skip integration_serial
Note:
-
Certain tests cannot be run in parallel with others, and these tests are typically in files postfixed with _serial. Running the entire set of tests for a package (e.g.,
cargo test -p readyset-server
) may fail if serial tests are included. -
Running tests may require increasing file descriptor limits. You can do so by running
ulimit -Sn 65535
.
When running ReadySet in a performance-critical setting, make sure you compile with the --release
flag.
ReadySet is licensed under the BSL 1.1 license, converting to the open-source Apache 2.0 license after 4 years. It is free to use on any number of nodes.