- Core dependencies:
NumPy
,SciPy
,PyBullet
,Pinocchio
,Gymnasium
,pyquaternion
,MuJoCo
- Dependencies to run the demos:
ProxSuite
,ndcurves
.
To install this package, first install the dependencies. NumPy
, MuJoCo
, SciPy
, PyBullet
, Pinocchio
, and ProxSuite
, ndcurves
, and Gymnasium
can be installed while running the setup code below. It is recommended to install everything within a conda environment. Then, run the following commands
git clone https://github.com/BolunDai0216/FR3Env.git
cd FR3Env
python3 -m pip install .
python examples/waypoint_hierarchical_proxqp_demo.py
fr3_demo.mp4
mjpython examples/hello_world_mj_diff_ik.py
fr3_mj_demo.mov
Cameras can be simulated and placed at various places in the environment. Our simulator provides two useful functions to make this task easiers. The first function converts the camera intrinsic paramters as found using ROS or OpenCV calibration procedure to a pybullet projection matrix and the other constructs the view matrix given the [R|t]
paris:
from FR3Env.utils import render_utils
'''
K is the camera matrix as estimated using ROS/OpenCV, w,h are image width and height
during calibration, and near, far indicate the interest depth to be rendered'''
render_utils.projectionMatrix = cvK2BulletP(K, w, h, near, far)
''' q and t are two lists representing the orientation and translation
as parameterized by ROS-TF conventions.'''
render_utils.viewMatrix = cvPose2BulletView(q, t)
Using the computed matrices above, you can grab images from the environment as follows:
import pybullet as p
_, _, rgb, depth, segmentation = p.getCameraImage(W, H, viewMatrix, projectionMatrix, shadow = True)
The function above returns the undistorted images, segmentation, and depth maps.
- To make
pinocchio
work all of the links have to have a specified mass and inertia.
The FR3 URDF file is generated using fr3-urdf-pybullet
To cite FR3Env in your academic research, please use the following bibtex entry:
@article{DaiKKGTK23,
author = {Bolun Dai and Rooholla Khorrambakht and Prashanth Krishnamurthy and Vin{\'{\i}}cius Gon{\c{c}}alves and Anthony Tzes and Farshad Khorrami},
title = {Safe Navigation and Obstacle Avoidance Using Differentiable Optimization Based Control Barrier Functions},
journal = {{IEEE} Robotics and Automation Letters},
year = {2023},
volume = {8},
number = {9},
pages = {5376-5383},
}
When running the MuJoCo viewer on MacOS, you may encounter the following error:
OSError: dlopen(/System/Library/OpenGL.framework/OpenGL, 0x0006): tried: '/System/Library/OpenGL.framework/OpenGL' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/System/Library/OpenGL.framework/OpenGL' (no such file), '/System/Library/OpenGL.framework/OpenGL' (no such file, not in dyld cache)
If you are using a conda environment, you can fix this by going to (replace 3.8
with your python version)
cd $CONDA_PREFIX/lib/python3.8/site-packages/OpenGL/platform
and change the line of _loadLibraryWindows()
in ctypesloader.py
from
fullName = util.find_library( name )
to
fullName = '/System/Library/Frameworks/OpenGL.framework/OpenGL'
For more details, see here.