Merge branch 'dev'

.github/workflows/pythonpackage.yml

@@ -12,7 +12,7 @@ jobs:
 
  strategy:
  matrix:
- python-version: [3.5, 3.6, 3.7, 3.8]
+ python-version: [3.6, 3.7, 3.8, 3.9]
  os: [ubuntu-latest, windows-latest, macOS-latest]
 
  steps:

CHANGES.md

@@ -1,6 +1,23 @@
 Revision History
 ================
 
+v0.8.0 (Nov 12, 2020)
+---------------------
+- New animation controller that synchronizes animations, audio playback, and image displaying.
+- Procedural face generation to bring the roboto to life.
+- Loading of Cozmo resource files - activities, behaviors, emotions, light animations
+ (thanks to Aitor Miguel Blanco / gimait)
+- pycozmo.audiokinetic module for working with Audiokinetic WWise SoundBank files.
+- New tool for managing Cozmo resources - pycozmo_resources.py .
+- Initial Cozmo application with rudimentary reactions and behaviors - pycozmo_app.py .
+- Cozmo protocol client robustness improvements.
+- CLAD encoding optimizations.
+- Cliff detection and procedural face rendering improvements (thanks to Aitor Miguel Blanco / gimait)
+- Replaced pycozmo.run_program() with pycozmo.connect() context manager. 
+- Renamed the NextFrame packet to OutputSilence to better describe its function.
+- Dropped support for Python 3.5 and added support for Python 3.9.
+- Bug fixes and documentation improvements.
+
 v0.7.0 (Sep 26, 2020)
 ---------------------
 - Full robot audio support and a new AudioManager class (thanks to Aitor Miguel Blanco / gimait). 

README.md

@@ -2,32 +2,33 @@
 PyCozmo
 =======
 
-`PyCozmo` is a pure-Python [Cozmo robot](https://www.digitaldreamlabs.com/pages/cozmo) communication library.
-It allows controlling a Cozmo robot directly, without having to go through a mobile device, running the Cozmo app.
+`PyCozmo` is a pure-Python communication library, alternative SDK, and application for the
+[Cozmo robot](https://www.digitaldreamlabs.com/pages/cozmo) . It allows controlling a Cozmo robot directly, without
+having to go through a mobile device, running the Cozmo app.
 
 The library is loosely based on the [Anki Cozmo Python SDK](https://github.com/anki/cozmo-python-sdk) and the
 [cozmoclad](https://pypi.org/project/cozmoclad/) ("C-Like Abstract Data") library.
 
-This project is a tool for exploring the hardware and software of Digital Dream Labs (originally Anki) Cozmo robot.
+This project is a tool for exploring the hardware and software of the Digital Dream Labs (originally Anki) Cozmo robot.
 It is unstable and heavily under development.
 
 
 Usage
 -----
 
 Basic:
+
 ```python
 import time
 import pycozmo
 
-def pycozmo_program(cli):
+with pycozmo.connect() as cli:
  cli.set_head_angle(angle=0.6)
  time.sleep(1)
-
-pycozmo.run_program(pycozmo_program)
 ```
 
 Advanced:
+
 ```python
 import pycozmo
 
@@ -60,44 +61,73 @@ Sensors:
 - [x] Battery voltage
 - [x] Cube battery voltage
 - [x] Cube accelerometers
+- [x] Backpack button (v1.5 hardware and newer)
 
 Actuators:
 - [x] Wheel motors
 - [x] Head motor
 - [x] Lift motor
-- [x] Backpack LEDs
-- [x] IR LED
 - [x] OLED display
 - [x] Speaker
+- [x] Backpack LEDs
+- [x] IR LED
 - [x] Cube LEDs
-- [x] Platform LEDs
+- [x] Platform LEDs (when available)
 
-Communication:
+On-board functions (see [docs/functions.md](docs/functions.md) for details:
 - [x] Wi-Fi AP
 - [x] Bluetooth LE
-
-Storage:
-- [x] NVRAM
-- [x] Firmware update
-
-Functions:
 - [x] Localization
 - [x] Path tracking
-- [x] Procedural faces
-- [x] Animations from FlatBuffers .bin files
+- [x] NV RAM storage
+- [x] Over-the-air (OTA) firmware updates
+
+Off-board functions (see [docs/offboard_functions.md](docs/offboard_functions.md) for details:
+- [x] Procedural face generation
+- [x] Cozmo animations from FlatBuffers .bin files
+- [ ] Personality engine - work in progress
+- [ ] Cozmo behaviors - work in progress
+- [ ] Motion detection
+- [ ] Object (cube and platform) detection
+- [ ] Cube marker recognition
+- [ ] Face detection
+- [ ] Face recognition
+- [ ] Facial expression estimation
+- [ ] Pet detection
+- [ ] Camera calibration
+- [ ] Navigation map building
+- [ ] Text-to-speech
+- [ ] Songs
+
+Extra off-board functions:
+- [ ] Vector animations from FlatBuffers .bin files
+- [ ] Vector behaviors
+- [ ] ArUco marker recognition
+- [ ] Cozmo and Vector robot detection
+- [ ] Drivable area estimation
+- [ ] Voice commands
+
+If you have ideas for other functionality [share them via GitHub](https://github.com/zayfod/pycozmo/issues).
 
 
 Tools
 -----
 
+- [pycozmo_app.py](tools/pycozmo_app.py) - an alternative Cozmo application, implementing off-board functions
+ ([video](https://youtu.be/gMEc6RzIm-E)).
 - [pycozmo_dump.py](tools/pycozmo_dump.py) - a command-line application that can read and annotate Cozmo communication
  from [pcap files](https://en.wikipedia.org/wiki/Pcap) or capture it live using
  [pypcap](https://github.com/pynetwork/pypcap).
 - [pycozmo_replay.py](tools/pycozmo_replay.py) - a basic command-line application that can replay .pcap files back to
  Cozmo.
 - [pycozmo_anim.py](tools/pycozmo_anim.py) - a tool for examining and manipulating animation files.
 - [pycozmo_update.py](tools/pycozmo_update.py) - a tool for over-the-air (OTA) updates of Cozmo's firmware.
+- [pycozmo_protocol_generator.py](tools/pycozmo_protocol_generator.py) - a tool for generating Cozmo protocol encoder
+ code.
 
+**Note**: PyCozmo and `pycozmo_protocol_generator.py` in particular could be used as a base for creating a Cozmo protocol
+encoder code generator for languages other than Python (C/C++, Java, etc.).
+
 
 Examples
 --------
@@ -115,6 +145,7 @@ Basic:
 Advanced:
 - [display_lines.py](examples/display_lines.py) - demonstrates 2D graphics, using
  [PIL.ImageDraw](https://pillow.readthedocs.io/en/stable/reference/ImageDraw.html) on Cozmo's display
+ ([video](https://youtu.be/ZT81PlmItrU))
 - [rc.py](examples/rc.py) - turns Cozmo into an RC tank that can be driven with an XBox 360 Wireless controller or 
  Logitech Gamepad F310
 - [video.py](examples/video.py) - demonstrates visualizing video captured from the camera back on display
@@ -124,7 +155,9 @@ Advanced:
 - [audio.py](examples/audio.py) - demonstrates 22 kHz, 16-bit, mono WAVE file playback through Cozmo's speaker 
 - [nvram.py](examples/nvram.py) - demonstrates reading data from Cozmo's NVRAM (non-volatile memory)
 - [procedural_face.py](examples/procedural_face.py) - demonstrates drawing a procedural face on Cozmo's display
-- [procedural_face_show.py](examples/procedural_face_show.py) - demonstrates generating a procedural face 
+- [procedural_face_show.py](examples/procedural_face_show.py) - demonstrates generating a procedural face
+- [procedural_face_expressions.py](examples/procedural_face_expressions.py) - demonstrates displaying various facial
+ expressions ([video](https://youtu.be/UyMTEOG2FyU))
 - [anim.py](examples/anim.py) - demonstrates animating Cozmo
 
 
@@ -171,24 +204,10 @@ the low-level UDP communication with Cozmo.
 ```
 
 
-Limitations
------------
-
-- some high-level Cozmo SDK features are implemented in the Cozmo app and have no equivalent in PyCozmo, today:
- - personality engine
- - behaviors
- - motion detection
- - face detection
- - facial expression estimation
- - text-to-speech
- - songs
-- there is no Wi-Fi control. The library assumes a Wi-Fi connection to Cozmo, established in advance.
-
-
 Requirements
 ------------
 
-- Python 3.5.4
+- [Python](https://www.python.org/downloads/) 3.6.0 or newer
 - [Pillow](https://github.com/python-pillow/Pillow) 6.0.0 - Python image library
 - [FlatBuffers](https://github.com/google/flatbuffers) - serialization library
 - [dpkt](https://github.com/kbandla/dpkt) - TCP/IP packet parsing library 
@@ -201,6 +220,8 @@ Using pip:
 
 ```
 pip install --user pycozmo
+
+pycozmo_resources.py download
 ```
 
 From source:
@@ -209,6 +230,8 @@ From source:
 git clone https://github.com/zayfod/pycozmo.git
 cd pycozmo
 python setup.py install --user
+
+pycozmo_resources.py download
 ```
 
 From source, for development:
@@ -218,6 +241,8 @@ git clone [email protected]:zayfod/pycozmo.git
 cd pycozmo
 python setup.py develop --user
 pip install --user -r requirements-dev.txt
+
+pycozmo_resources.py download
 ```
 
 

docs/architecture.md

@@ -0,0 +1,142 @@
+
+PyCozmo Architecture
+====================
+
+
+Overview
+--------
+
+PyCozmo is designed as a multithreaded library.
+
+It is organized in three layers with each higher layer building on the ones below it: 
+- low-level connection layer
+- client or SDK layer
+- application layer
+
+Each layer provides it's own API and can be used independently.
+
+The following diagram illustrates the library architecture.
+
+```
+ ^
+ |
+ +-----------------------------------------+ |
+ | | |
+ | Brain | |
+ | | |
+ | | |
+ +-----+--------------+--------------+-----+ |
+ ^ | ^ |
+ | | | |
+ +-----------+ +-----+-----+ | | | Application
+ | | | | | | | Layer
+ | Heartbeat | | Reaction | | | |
+ | Thread | | Thread | | | |
+ | | | | | | |
+ +-----+-----+ +-----+-----+ | | |
+ | ^ | Commands | Events |
+ | | | | |
+ | +-+ | | |
+ | |-| Event | | |
+ | |-| Queue | | |
+ | +-+ | | |
+ | ^ | | |
+ | | | | v
+ +------------->+ Reactions | |
+ | v | ^
+ +-----+--------------+--------------+-----+ |
+ | | |
+ | Client | |
+ | | |
+ | | |
+ +-----+--------------+--------------+-----+ |
+ | | ^ | SDK
+ v | | | Layer
+ +-+ | | |
+ Animation |-| | Commands | Events |
+ Queue |-| | | |
+ +-+ | | |
+ | | | |
+ v v | |
++-----------+ +-----+-----+ +-----+--------------+-----+ |
+| | | | | | |
+| Face | | Animation | | Connection | v
+| Generator +------>+ Thread | | Thread |
+| | | | | | ^
++-----------+ +-----+-----+ +-----+--------------+-----+ |
+ | | ^ |
+ +------------->+ | |
+ | | |
+ v | |
+ +-+ +-+ |
+ Outgoing |-| |-| Incoming |
+ Message |-| |-| Message |
+ Queue +-+ +-+ Queue |
+ | ^ |
+ v | | Connection
+ +-----+-----+ +-----+-----+ | Layer
+ | | | | |
+ | Send | | Receive | |
+ | Thread | | Thread | |
+ | | | | |
+ +---------+-+ +--+--------+ |
+ | ^ |
+ | | |
+ | | |
+ v | |
+ +-+-------+-+ |
+ | | |
+ | UDP | |
+ | Socket | |
+ | | |
+ +-----------+ |
+ v
+```
+
+Connection Layer
+----------------
+
+The connection layer implements the Cozmo communication protocol.
+
+The receive thread reads Cozmo protocol frames, encapsulated in UDP datagrams, from the UDP socket. It maintains
+a receive window for incoming packets and sends a stream of incoming packets in the correct order over the incoming
+message queue to the connection thread.
+
+The send thread reads a stream of outgoing packets from the outgoing message queue, builds Cozmo protocol frames
+and sends them over the UDP socket. It maintains a send window and resends packets that are not acknowledged in time.
+
+The connection thread reads a stream of incoming packets from the incoming message queue and dispatches them to
+registered handler functions. It sends ping packets on a regular basis to maintain connection with the robot. 
+
+
+Client Layer (SDK)
+------------------
+
+The client layer provides access to robot on-board functions.
+
+It allows sending commands and registering handler function for incoming packets and events.
+
+It performs:
+- camera image reconstruction
+- display image encoding
+- audio encoding
+- animation and audio playback
+- procedural face generation 
+
+The animation controller synchronizes animations, audio playback, and image display. It works as a separate thread
+that aims to send images and audio to the robot at 30 frames per second. All on-board function of the robot are
+synchronized to this framerate, including images, audio playback, backpack and cube LED animations.
+
+
+Application Layer
+-----------------
+
+The application layer implements high-level off-board functions:
+- reactions and behaviors
+- personality engine
+- computer vision (CV) camera image processing
+
+Events from the client layer are converted to reactions. The reaction thread reads events from its incoming event
+queue and handles them appropriately. Reactions normally trigger behaviors.
+
+The heartbeat thread drives the personality engine and timers for activities and behaviors.