Run LLaMA on your computer.

Github Twitter Discord

1. Powered by llama.cpp and llama-dl CDN
2. Hackable web app included
3. Ships with JavaScript API
4. Ships with Socket.io API

Dalai runs on all of the following operating systems:

1. Linux
2. Mac
3. Windows

Runs on most modern computers. Unless your computer is very very old, it should work.

You need a lot of space for storing the models.

You do NOT have to install all models, you can install one by one. Let's take a look at how much space each model takes up:

NOTE

The following numbers assume that you DO NOT touch the original model files and keep BOTH the original model files AND the quantized versions.

You can optimize this if you delete the original models (which are much larger) after installation and keep only the quantized versions.

• Full: The model takes up 31.17GB
• Quantized: 4.21GB

• Full: The model takes up 60.21GB
• Quantized: 4.07GB * 2 = 8.14GB

• Full: The model takes up 150.48GB
• Quantized: 5.09GB * 4 = 20.36GB

• Full: The model takes up 432.64GB
• Quantized: 5.11GB * 8 = 40.88GB

Install Node.js

Basic install (7B model only)

npx dalai llama

Or, install all models

npx dalai llama 7B 13B 30B 65B

The install command :

1. Creates a folder named dalai under your home directory (~)
2. Installs and builds the llama.cpp project under ~/llama.cpp
3. Downloads all the requested models from the llama-dl CDN to ~/llama.cpp/models
4. Runs some tasks to convert the LLaMA models so they can be used

After everything has been installed, launch the Web UI with:

npx dalai serve

Now open https://localhost:3000 in your browser. Have fun!

On windows, you need to install Visual Studio before installing Dalai.

Press the button below to visit the Visual Studio downloads page and download:

Download Microsoft Visual Studio

IMPORTANT!!!

When installing Visual Studio, make sure to check the 3 options as highlighted below:

1. Python development
2. Node.js development
3. Desktop development with C++

Basic install (7B model only)

npx dalai llama

Or, install all models

npx dalai llama 7B 13B 30B 65B

The install command :

1. Creates a folder named dalai under your home directory (~)
2. Installs and builds the llama.cpp project under ~/llama.cpp
3. Downloads all the requested models from the llama-dl CDN to ~/llama.cpp/models
4. Runs some tasks to convert the LLaMA models so they can be used

If this worked without any errors, go to step 3.

Ohterwise try the troubleshoot below:

In case above steps fail to install, try installing node.js and python separately.

Install Python:

Download Python

Install Node.js:

Download Node.js

After both have been installed, open powershell and type python to see if the application exists. And also type node to see if the application exists as well.

Once you've checked that they both exist, try the npx dalai llama command again.

After everything has been installed, launch the Web UI with:

npx dalai serve

Now open https://localhost:3000 in your browser. Have fun!

You need to make sure you have the correct version of Python and Node.js installed.

Make sure the version is 3.10 or lower (not 3.11)

Python must be 3.10 or below (pytorch and other libraries are not supported yet on the latest)

Download node.js

Make sure the version is 18 or higher

Download node.js

Basic install (7B model only)

npx dalai llama

Or, install all models

npx dalai llama 7B 13B 30B 65B

The install command :

1. Creates a folder named dalai under your home directory (~)
2. Installs and builds the llama.cpp project under ~/llama.cpp
3. Downloads all the requested models from the llama-dl CDN to ~/llama.cpp/models
4. Runs some tasks to convert the LLaMA models so they can be used

After everything has been installed, launch the Web UI with:

npx dalai serve

Now open https://localhost:3000 in your browser. Have fun!

Dalai is also an NPM package:

1. programmatically install
2. locally make requests to the model
3. run a dalai server (powered by socket.io)
4. programmatically make requests to a remote dalai server (via socket.io)

Dalai is an NPM package. You can install it using:

npm install dalai

const dalai = new Dalai(home)

• home: (optional) manually specify the llama.cpp folder

By default, Dalai automatically stores the entire llama.cpp repository under ~/llama.cpp.

However, often you may already have a llama.cpp repository somewhere else on your machine and want to just use that folder. In this case you can pass in the home attribute.

Creates a workspace at ~/llama.cpp

const dalai = new Dalai()

Manually set the llama.cpp path:

const dalai = new Dalai("/Documents/llama.cpp")

dalai.request(req, callback)

• req: a request object. made up of the following attributes:
  • prompt: (required) The prompt string
  • model: (required) The model name to query ("7B", "13B", etc.)
  • url: only needed if connecting to a remote dalai server
    • if unspecified, it uses the node.js API to directly run dalai locally
    • if specified (for example ws:https://localhost:3000) it looks for a socket.io endpoint at the URL and connects to it.
  • threads: The number of threads to use (The default is 8 if unspecified)
  • n_predict: The number of tokens to return (The default is 128 if unspecified)
  • seed: The seed. The default is -1 (none)
  • top_k
  • top_p
  • repeat_last_n
  • repeat_penalty
  • temp: temperature
  • batch_size: batch size
  • skip_end: by default, every session ends with \n\n<end>, which can be used as a marker to know when the full response has returned. However sometimes you may not want this suffix. Set skip_end: true and the response will no longer end with \n\n<end>
• callback: the streaming callback function that gets called every time the client gets any token response back from the model

Using node.js, you just need to initialize a Dalai object with new Dalai() and then use it.

const Dalai = require('dalai')
new Dalai().request({
  model: "7B",
  prompt: "The following is a conversation between a boy and a girl:",
}, (token) => {
  process.stdout.write(token)
})

To make use of this in a browser or any other language, you can use thie socket.io API.

First you need to run a Dalai socket server:

// server.js
const Dalai = require('dalai')
new Dalai().serve(3000)     // port 3000

Then once the server is running, simply make requests to it by passing the ws:https://localhost:3000 socket url when initializing the Dalai object:

const Dalai = require("dalai")
new Dalai().request({
  url: "ws:https://localhost:3000",
  model: "7B",
  prompt: "The following is a conversation between a boy and a girl:",
}, (token) => {
  console.log("token", token)
})

Starts a socket.io server at port

dalai.serve(port)

const Dalai = require("dalai")
new Dalai().serve(3000)

connect with an existing http instance (The http npm package)

dalai.http(http)

• http: The http object

This is useful when you're trying to plug dalai into an existing node.js web app

const app = require('express')();
const http = require('http').Server(app);
dalai.http(http)
http.listen(3000, () => {
  console.log("server started")
})

await dalai.install(model1, model2, ...)

• models: the model names to install ("7B"`, "13B", "30B", "65B", etc)

Install the "7B" and "13B" models:

const Dalai = require("dalai");
const dalai = new Dalai()
await dalai.install("7B", "13B")

returns the array of installed models

const models = await dalai.installed()

const Dalai = require("dalai");
const dalai = new Dalai()
const models = await dalai.installed()
console.log(models)     // prints ["7B", "13B"]

Dalai is a young project and will evolve quickly.

To update dalai, you will need to run the dalai command with a version number specified (You only need to do this once when you update).

For example, let's say you've been using [email protected] but a new version [email protected] came out.

The simplest way to update is to just run the dalai server:

npx [email protected] serve

Once you run the command it will ask you if you want to update. Confirm, and it will now install 0.2.0, and from that point on you don't need to specify the version. You can just run npx dalai serve and the new version will be executed from that point on.

Have questions or feedback? Follow the project through the following outlets:

Github Twitter Discord