People with learning disabilities can encounter difficulty in expressing themselves in speech. Makaton is a language programme that uses signs and symbols to enable users of all ages to communicate effectively. WALDO is a machine learning-enabled assisted living device meant for use by Makaton users and their carers in a care home setting. WALDO’s capabilities include:
- Makaton sign language interpretation using machine learning
- Easy-to-use interface, and a cute and endearing package to encourage use by people with learning disabilities and their carers
- Programmable buttons for users to quickly express pre-set phrases or emotions
WALDO is intended for use in a care home setting in the following ways:
Makaton sign interpretation
- Makaton user (care home patient) signs to WALDO
- WALDO uses machine learning to interpret sign
- WALDO vocalises sign for carer to understand what user intends to express
- Carer reacts accordingly, completing the interaction between the user and carer This implementation enables the carer to understand the user without having to undergo the time-consuming process of learning Makaton.
Expression of pre-set phrases/emotions
- User (by him/herself, or via carer) sets common phrases that he/she uses frequently to correspond to physical buttons on WALDO
- User labels buttons accordingly for ease of use
- User presses the relevant button, WALDO vocalises pre-set phrase
- Carer responds accordingly This implementation enables people with learning disabilities that have difficulty expressing themselves in speech to easily express everyday sentiments.
This is an overview of the various sections and subsections of WALDO's Virtual Design History File (VDHF). The VDHF details the developmental process of WALDO, the necessary steps to recreate it and the ethical and sustainability considerations of WALDO.
| Section | Subsection | Description |
|---|---|---|
| Overview | Overview of WALDO, contents of Virtual Design History File | |
| 1. Setup | 1.1 Jetson Nano Setup | Jetson Nano Setup Guide |
| 1.2 Pi Setup | Raspberry Pi Setup Guide | |
| 1.3 Hardware Setup | Diagram of device hardware connections | |
| 2. Machine Learning | 2.1 Machine Learning | Details model design history and evolution as well as implementation on the edge device |
| 2.2 Server Environment Setup | Setting up server environment for further model training | |
| Server Training Code | Code to be run on server for model training | |
| Jetson Nano Execution Code | Code to be run on Jetson Nano for Makaton sign recognition | |
| Model and Weights | Saved weights of the final trained model | |
| 3. Raspberry Pi | 3.1 Overview of Pi Function | Outline of functions and connections of Pi |
| Pi Execution Code | Code to be run on Raspberry Pi for high level tasks and audio generations through the IBM Watson text to speech module | |
| 4. Hardware | 4.1 3D printing files | Contains Standard Tessellation Language (STL) files of the Jetson Nano case, the Pi case, the button support structure and WALDO’s eyepiece |
| 4.2 Power Supply | Summary of considerations when choosing WALDO’s power supply | |
| 4.3 Hardware List | List of all hardware components used in assembly of WALDO | |
| 5. Administrative | 5.1 Record of Meetings | Meeting minutes and decisions made |
| 5.2 Gantt Chart | Intended timeline of project (plotted during first meeting at IBM Hursley) | |
| 5.3 Bill of Materials | Record of expenditure and outline of cost of device | |
| 5.4 Data Collection | Outline of procedure, considerations and product of data collection | |
| 6. Miscellaneous | 6.1 Ethical Considerations | Outline of ethical considerations in the design, building and use of WALDO |
| 6.2 Sustainability considerations | Outline of sustainability considerations in the design, building and use of WALDO | |
| 6.3 Blog | Rationale behind the blog and links to the blog posts | |
| 6.4 Future Work | Discussion of present implementation and possible extensions to the project | |
| 6.5 Leaflet | Informational leaflet on WALDO | |
| 6.6 Poster | Poster for hackbooth and presentation | |
The WALDO team consists of 5 students from the Electrical Engineering (EE) Department of Imperial College London. The key tasks performed by each member is as follows,
- Joshua Chan : Administrative, Data Collection, Peripheral Sensors Implementation
- Lua Ying Hao : Jetson Nano Implementation, Machine Learning
- Ng Yi Song : Data Collection, Peripheral Sensors Implementation
- Patrick John Chia : Jetson Nano Implementation, Machine Learning
- Yeow En Kai Joel : Administrative, Data Collection and Processing, Peripheral Sensors Implementation
We would like to thank:
- Dr Steve Wright, for his invaluable guidance and encouragement throughout the project
- Louise Cooper, John McNamara & Emily Larkin from IBM for their clear direction and unwavering support in the implementation of this project
- Precious Homes, for supporting this project by providing valuable insight into the use of Makaton, as well as training videos
- Vic, May & Amine at the teaching labs, for their untiring technical support
- All who volunteered their time to help us in the collection of training data