KerasCV is a library of modular computer vision components that work natively with TensorFlow, JAX, or PyTorch. Built on Keras Core, these models, layers, metrics, callbacks, etc., can be trained and serialized in any framework and re-used in another without costly migrations. See "Using KerasCV with Keras Core" below for more details on multi-framework KerasCV.
KerasCV can be understood as a horizontal extension of the Keras API: the components are new first-party Keras objects that are too specialized to be added to core Keras. They receive the same level of polish and backwards compatibility guarantees as the core Keras API, and they are maintained by the Keras team.
Our APIs assist in common computer vision tasks such as data augmentation, classification, object detection, segmentation, image generation, and more. Applied computer vision engineers can leverage KerasCV to quickly assemble production-grade, state-of-the-art training and inference pipelines for all of these common tasks.
- List of available models and presets
- Developer Guides
- Contributing Guide
- Call for Contributions
- API Design Guidelines
To install the latest official release:
pip install keras-cv tensorflow --upgrade
To install the latest unreleased changes to the library, we recommend using pip to install directly from the master branch on github:
pip install git+https://github.com/keras-team/keras-cv.git tensorflow --upgrade
As of version 0.6.0
, KerasCV supports multiple backends with Keras Core out of
the box. There are two ways to configure KerasCV to run with multi-backend
support:
- Via the
KERAS_BACKEND
environment variable. If set, then KerasCV will be using Keras Core with the backend specified (e.g.,KERAS_BACKEND=jax
). - Via the
.keras/keras.json
and.keras/keras_cv.json
config files (which are automatically created the first time you import KerasCV):- Set your backend of choice in
.keras/keras.json
; e.g.,"backend": "jax"
. - Set
"multi_backend": True
in.keras/keras_cv.json
.
- Set your backend of choice in
Once that configuration step is done, you can just import KerasCV and start using it on top of your backend of choice:
import keras_cv
import keras_core as keras
filepath = keras.utils.get_file(origin="https://i.imgur.com/gCNcJJI.jpg")
image = np.array(keras.utils.load_img(filepath))
image_resized = keras.ops.image.resize(image, (640, 640))[None, ...]
model = keras_cv.models.YOLOV8Detector.from_preset(
"yolo_v8_m_pascalvoc",
bounding_box_format="xywh",
)
predictions = model.predict(image_resized)
Until Keras Core is officially released as Keras 3.0, KerasCV will use
tf.keras
as the default backend. To restore this default behavior, simply
unset KERAS_BACKEND
and ensure that "multi_backend": False
or is unset in
.keras/keras_cv.json
. You will need to restart the Python runtime for changes
to take effect.
import tensorflow as tf
import keras_cv
import tensorflow_datasets as tfds
import keras_core as keras
# Create a preprocessing pipeline with augmentations
BATCH_SIZE = 16
NUM_CLASSES = 3
augmenter = keras_cv.layers.Augmenter(
[
keras_cv.layers.RandomFlip(),
keras_cv.layers.RandAugment(value_range=(0, 255)),
keras_cv.layers.CutMix(),
],
)
def preprocess_data(images, labels, augment=False):
labels = tf.one_hot(labels, NUM_CLASSES)
inputs = {"images": images, "labels": labels}
outputs = inputs
if augment:
outputs = augmenter(outputs)
return outputs['images'], outputs['labels']
train_dataset, test_dataset = tfds.load(
'rock_paper_scissors',
as_supervised=True,
split=['train', 'test'],
)
train_dataset = train_dataset.batch(BATCH_SIZE).map(
lambda x, y: preprocess_data(x, y, augment=True),
num_parallel_calls=tf.data.AUTOTUNE).prefetch(
tf.data.AUTOTUNE)
test_dataset = test_dataset.batch(BATCH_SIZE).map(
preprocess_data, num_parallel_calls=tf.data.AUTOTUNE).prefetch(
tf.data.AUTOTUNE)
# Create a model using a pretrained backbone
backbone = keras_cv.models.EfficientNetV2Backbone.from_preset(
"efficientnetv2_b0_imagenet"
)
model = keras_cv.models.ImageClassifier(
backbone=backbone,
num_classes=NUM_CLASSES,
activation="softmax",
)
model.compile(
loss='categorical_crossentropy',
optimizer=keras.optimizers.Adam(learning_rate=1e-5),
metrics=['accuracy']
)
# Train your model
model.fit(
train_dataset,
validation_data=test_dataset,
epochs=8,
)
If you'd like to contribute, please see our contributing guide.
To find an issue to tackle, please check our call for contributions.
We would like to leverage/outsource the Keras community not only for bug reporting, but also for active development for feature delivery. To achieve this, here is the predefined process for how to contribute to this repository:
- Contributors are always welcome to help us fix an issue, add tests, better documentation.
- If contributors would like to create a backbone, we usually require a pre-trained weight set with the model for one dataset as the first PR, and a training script as a follow-up. The training script will preferably help us reproduce the results claimed from paper. The backbone should be generic but the training script can contain paper specific parameters such as learning rate schedules and weight decays. The training script will be used to produce leaderboard results. Exceptions apply to large transformer-based models which are difficult to train. If this is the case, contributors should let us know so the team can help in training the model or providing GCP resources.
- If contributors would like to create a meta arch, please try to be aligned with our roadmap and create a PR for design review to make sure the meta arch is modular.
- If contributors would like to create a new input formatting which is not in our roadmap for the next 6 months, e.g., keypoint, please create an issue and ask for a sponsor.
- If contributors would like to support a new task which is not in our roadmap for the next 6 months, e.g., 3D reconstruction, please create an issue and ask for a sponsor.
Thank you to all of our wonderful contributors!
Many models in KerasCV come with pre-trained weights. With the exception of StableDiffusion and the standard Vision Transformer, all of these weights are trained using Keras and KerasCV components and training scripts in this repository. While some models are not trained with the same parameters or preprocessing pipeline as defined in their original publications, the KerasCV team ensures strong numerical performance. Performance metrics for the provided pre-trained weights can be found in the training history for each documented task. An example of this can be found in the ImageNet classification training history for backbone models. All results are reproducible using the training scripts in this repository.
Historically, many models have been trained on image datasets rescaled via manually
crafted normalization schemes.
The most common variant of manually crafted normalization scheme is subtraction of the
imagenet mean pixel followed by standard deviation normalization based on the imagenet
pixel standard deviation.
This scheme is an artifact of the days of manual feature engineering, but is no longer
required to score state of the art scores using modern deep learning architectures.
Due to this, KerasCV is standardized to operate on images that have been rescaled using
a simple 1/255
rescaling layer.
This can be seen in all KerasCV training pipelines and code examples.
Note that in some of the 3D Object Detection layers, custom TF ops are used. The binaries for these ops are not shipped in our PyPi package in order to keep our wheels pure-Python.
If you'd like to use these custom ops, you can install from source using the instructions below.
Installing custom ops from source requires the Bazel build system (version >= 5.4.0). Steps to install Bazel can be found here.
git clone https://github.com/keras-team/keras-cv.git
cd keras-cv
python3 build_deps/configure.py
bazel build build_pip_pkg
export BUILD_WITH_CUSTOM_OPS=true
bazel-bin/build_pip_pkg wheels
pip install wheels/keras_cv-*.whl
Note that GitHub actions exist to release KerasCV with custom ops, but are currently disabled. You can use these actions in your own fork to create wheels for Linux (manylinux2014), MacOS (both x86 and ARM), and Windows.
KerasCV provides access to pre-trained models via the keras_cv.models
API.
These pre-trained models are provided on an "as is" basis, without warranties
or conditions of any kind.
The following underlying models are provided by third parties, and are subject to separate
licenses:
StableDiffusion, Vision Transfomer
If KerasCV helps your research, we appreciate your citations. Here is the BibTeX entry:
@misc{wood2022kerascv,
title={KerasCV},
author={Wood, Luke and Tan, Zhenyu and Stenbit, Ian and Bischof, Jonathan and Zhu, Scott and Chollet, Fran\c{c}ois and others},
year={2022},
howpublished={\url{https://github.com/keras-team/keras-cv}},
}