Bigbluebutton already has live captioning for its meetings but Text-Track-Service is a project to help bigbluebutton auto-generate captions for their videos using multiple paid and free services. This project aims to make the software more accessible to students/users with hearing disabilities. This makes it easier for students/users that want to go back to a class/meeting and see the video again with captions. The documentation is for BigBlueButton admins that want to set up the text-track-service on a server already running with bigbluebutton to generate captions.
You can try it out here:- https://demo.bigbluebutton.org/
The video player will have a cc button. Click it and you should see captions for your video.
Here is a simple diagram of how it works:-
In this installation we will show you how to set up the Text-Track application piece of the puzzle & also how to create and use your account for the services you want.
You should have ssh access to a server with BigBlueButton already installed.
We need to first set up a separate user for our text-track service.
Steps to set up
adduser texttrack
usermod -aG sudo texttrack
su texttrack (switch to texttrack user)
Steps to test
sudo ls -la /root
Now we need to set up docker to make maintaining and updating the application easier.
Check if you have docker installed
docker --version
If you don't have docker installed please follow the steps given at Docker install for more information.
Steps to test
sudo docker run hello-world
Docker-compose is again just easier maintenance.
Steps to set up Follow instructions at:- Docker-compose install
Steps to test
docker-compose --version
Adding the texttrack user to docker will allow texttrack to access the files created by docker.
sudo usermod -a -G docker texttrack
We need to install git in order to clone our application from Github.
Steps to set up
sudo apt update
sudo apt install git
Steps to test
git --version
Now that we are set up with what we need, we can now clone the application from the github repo. Clone repo
cd /var
sudo mkdir docker
sudo chown texttrack:textrack /var/docker
cd docker
git clone https://github.com/bigbluebutton/text-track-service
cd text-track-service
The installation instructions are for IBM if you want to use another service visit this link: Google Docs services Follow instructions for setting up the service you want.
Steps to set up:
sign up to IBM at: Ibm sign-up After signing in you should be able to see the dashboard. Click on Watson in the left menu. Click on get started (convert audio into text) then you can give service name and/or tag name here. And select the blue create button in the right panel. You can give the service name of your choice. Then click create. Click on manage in the left panel and you will be able to see your credentials.
Save your apikey & url given.
In order to test your IBM credentials we have a audio_temp.flac file for you to test with in /var/docker/text-track-service/test_dir
Steps to test
once you have your IBM api_key & url open terminal and do the following: (make sure to replace {apikey} & {url} with your credentials)
cd /var/docker/text-track-service/test_dir
curl -X POST -u "apikey:{apikey}" \
--header "Content-Type: audio/flac" \
--data-binary @audio_temp.flac \
"{url}/v1/recognize"
Now that you have signed up to a service we need to tie that service with the application using your account. After you copy example-credentials as credentials it already has placeholders for all services based on what information is needed. Edit your information in and make sure you have all information need by that service.
Steps to set up
cp example-credentials.yaml credentials.yaml
Now edit your information in credentials.yaml (you can refer example-credentials.yaml file in /var/docker/text-track-service)
(extra step for google account)
create auth/google_auth_file.json and add file name to credentials.yaml (make sure google auth file owner is texttrack)
Now we want to set up the application to start, stop and restart by itself, so we are going to set up some systemd files to do that for us.
Steps to set up
cd /var/docker/text-track-service/systemd
sudo cp tts-docker.service /etc/systemd/system
sudo systemctl enable tts-docker
sudo chmod -R a+rX /var/docker/text-track-service/tmp/*
sudo systemctl start tts-docker
Steps to test
sudo journalctl -u tts-docker -f (see tailed logs)
Expected Result:
You should get the following result: ActiveRecord::NoDatabaseError: FATAL: database "myapp_development" does not exist
Now we need to fix the error we got for no database. In a new terminal launch the commands below to set up your database.
Steps to set up
cd /var/docker/text-track-service
sudo rm -R tmp/db
sudo chmod -R a+rX tmp/*
sudo chmod -R a+rX *
sudo docker-compose exec --user "$(id -u):$(id -g)" website rails db:create
sudo chmod -R a+rX tmp/*
sudo chmod -R a+rX *
sudo docker-compose exec --user "$(id -u):$(id -g)" website rails db:migrate
Add the following line to the end of the visudo file:
sudo visudo
texttrack ALL = NOPASSWD: /var/docker/text-track-service/deploy.sh
Save and close the file
cd /var/docker/text-track-service
./deploy.sh
Steps to test
sudo journalctl -u tts-docker -f
(You should no longer get a no db error)
Expected Result:
After creating databse now you should see lines that say your database has been created similar to:
(200.3ms) CREATE DATABASE "myapp_development" ENCODING = 'unicode'
↳ bin/rails:11
Created database 'myapp_development'
(103.8ms) CREATE DATABASE "myapp_test" ENCODING = 'unicode'
↳ bin/rails:11
Created database 'myapp_test'
After migrating database you should see output similar to:
(21.9ms) CREATE TABLE "schema_migrations" ("version" character varying NOT NULL PRIMARY KEY)
↳ bin/rails:11
(11.6ms) CREATE TABLE "ar_internal_metadata" ("key" character varying NOT NULL PRIMARY KEY, "value" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)
↳ bin/rails:11
(0.5ms) SELECT pg_try_advisory_lock(7864839756650184315)
↳ bin/rails:11
(0.7ms) SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
↳ bin/rails:11
Migrating to CreateCaptions (20190709150627)
(0.2ms) BEGIN
↳ bin/rails:11
== 20190709150627 CreateCaptions: migrating ===================================
-- create_table(:captions)
(11.5ms) CREATE TABLE "captions" ("id" bigserial primary key, "record_id" character varying, "service" character varying, "status" character varying, "caption_locale" character varying, "error" character varying, "bbb_url" character varying, "bbb_checksum" character varying, "kind" character varying, "label" character varying, "processtime" character varying, "start_time" character varying, "end_time" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)
↳ db/migrate/20190709150627_create_captions.rb:6
-> 0.0127s
== 20190709150627 CreateCaptions: migrated (0.0127s) ==========================
ActiveRecord::SchemaMigration Create (0.4ms) INSERT INTO "schema_migrations" ("version") VALUES ($1) RETURNING "version" [["version", "20190709150627"]]
↳ bin/rails:11
(1.9ms) COMMIT
↳ bin/rails:11
ActiveRecord::InternalMetadata Load (0.5ms) SELECT "ar_internal_metadata".* FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 LIMIT $2 [["key", "environment"], ["LIMIT", 1]]
↳ bin/rails:11
(0.1ms) BEGIN
↳ bin/rails:11
ActiveRecord::InternalMetadata Create (0.4ms) INSERT INTO "ar_internal_metadata" ("key", "value", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "key" [["key", "environment"], ["value", "development"], ["created_at", "2020-01-15 15:39:41.277333"], ["updated_at", "2020-01-15 15:39:41.277333"]]
↳ bin/rails:11
(1.1ms) COMMIT
↳ bin/rails:11
(0.4ms) SELECT pg_advisory_unlock(7864839756650184315)
↳ bin/rails:11
(0.3ms) SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
↳ bin/rails:11
The application needs some information about the BigBlueButton server to put the captions back onto the bbb server. Update the bigbluebutton.yml file with the needed information.
cd /usr/local/bigbluebutton/core/scripts/
To find your secret: bbb-conf -secret (shared secret)
Set up tts-secret in credentials.yaml and then enter that in /usr/local/bigbluebutton/core/scripts/bigbluebutton.yml
sudo vim bigbluebutton.yml
presentation_dir: /var/bigbluebutton/published/presentation
shared_secret: secret
temp_storage: /var/bigbluebutton/captions
tts_shared_secret: {tts-secret}
Now finally we need the bbb server to send the caption request to text-track application. This is done by the post publish file, we already have one set up for you. Copy it onto your server and your requests can now be sent to the text-track application from bbb server.
cd /usr/local/bigbluebutton/core/scripts/post_publish/
Make sure you have ffmpeg installed:
sudo apt-get install ffmpeg
Make sure you have the following gems installed:
cd /usr/local/bigbluebutton/core/scripts/post_publish/
sudo gem install rest-client
sudo gem install speech_to_text
sudo gem install jwt
(Replace your post_publish.rb with the one in /var/docker/text-track-service)
sudo cp /var/docker/text-track-service/post_publish.rb root@your_server:/usr/local/bigbluebutton/core/scripts/post_publish (use if bbb and text-track-service are running on different server)
sudo cp /var/docker/text-track-service/post_publish.rb /usr/local/bigbluebutton/core/scripts/post_publish (use if bbb and text-track-service are running on same server)
To change the service you are using in post_publish.rb just add service name to the end of the request url(deepspeech is default)
request = RestClient::Request.new(
method: :get,
url: "https://localhost:4000/caption/#{meeting_id}/en-US/",
payload: { :file => File.open("#{temp_storage}/#{meeting_id}/#{meeting_id}.wav", 'rb'),
:token => token }
)
eg. https://localhost:4000/caption/#{meeting_id}/en-US/google
eg. https://localhost:4000/caption/#{meeting_id}/en-US/ibm
eg. https://localhost:4000/caption/#{meeting_id}/en-US/speechmatics
eg. https://localhost:4000/caption/#{meeting_id}/en-US/threeplaymedia
eg. https://localhost:4000/caption/#{meeting_id}/en-US/deepspeech or https://localhost:4000/caption/#{meeting_id}/en-US/ (deepspeech is default)
Also we need to set up rap caption inbox worker as the application only drops the captions in the inbox folder. The rap caption inbox worker will be the one to move them from inbox to the presentation dir so your video can finally have captions.
Text-Track-Service only drops the files in the inbox folder at /var/bigbluebutton/captions/inbox
Now the rap-caption-inbox.rb should move it to the presentation dir(/var/bigbluebutton/published/presentation/<record-id>) to start it run the foll command:
cd /var/docker/text-track-service
sudo cp rap-caption-inbox.rb /var/bigbluebutton/captions/inbox/rap-caption-inbox.rb (copy file from repo)
cd /var/bigbluebutton/captions/inbox
sudo chown root:root /usr/local/bigbluebutton/core/scripts/rap-caption-inbox.rb
sudo chmod ugo+x /usr/local/bigbluebutton/core/scripts/rap-caption-inbox.rb
sudo systemctl start bbb-rap-caption-inbox.service
Finally make a recording/meeting on your server.
Look at the logs to make sure it processes successfully(sudo journalctl -u tts-docker -f)
Check the presentation folder of the record_id to see if a vtt file was generated. This can be found at (/var/bigbluebutton/published/presentation/<record_id>)
If there is a vtt file you have successfully transcribed your first meeting using IBM.
If you followed Step 7 your should be able to see tailed logs with the following command
sudo journalctl -u tts-docker -f
fix any errors shown and then re-deploy by running deploy.sh in the root folder
- If the logs do not show any errors but you are missing the vtt file in the presentation folder check the inbox folder at
/var/bigbluebutton/captions/inbox/(You should see a json and txt file.) - This means that the text track has done its job and the rap caption worker is not moving the files to the right location.
- To fix this you can copy the
rap-caption-inbox.rbfile from the repo to your bbb server at/usr/local/bigbluebutton/core/scripts/& make sure the owner is root. - Last step is make sure the
rap-caption-work.rbhas correct execute permissions if not just run the following command:
sudo chmod ugo+x /usr/local/bigbluebutton/core/scripts/rap-caption-inbox.rb
sudo systemctl start bbb-rap-caption-inbox.service
Here is a link to a google docs for signing up to the services: Google Docs services
To use other services you need to edit credentials.yaml file with the details required
Reference example-credentials.yaml for needed information
Finally edit post_publish.rb to use the new selected service as discussed in Step 10.
sudo vim /usr/local/bigbluebutton/core/scripts/post_publish/post_publish.rb
On line 113 add ibm or the service you want to (https://localhost:4000/caption/#{meeting_id}/en-US/) to the end of the line eg. https://localhost:4000/caption/#{meeting_id}/en-US/ibm
save & exit
To set up your own deepspeech server follow instructions at: Github deepspeech-web
We can now set up api commands that will give us all the information about the recordings processed by the application.
Install api commands
cd /var/docker/text-track-service/commands
./config.sh (run api config file)
you can now use the commands from anywhere in the terminal as long as you are ssh into the server
Info-Api usage:
| Command | Result |
|---|---|
| tts-all | shows list of all record-ids sent to the text-track-service |
| tts-processed | list of all successfully processed record-ids |
| tts-failed | list of all failed to process record-ids |
| tts-record <record_id> | get data for specific record_id |
| tts-delete <record_id> | delete data about a specific recording from text-track-service |
| tts-delete-all | delete all data about recordings from text-track-service |
We have also have an api endpoint that makes it possible to download a vtt file, audio file for a specific recording. After this you can edit the vtt(caption) file and reupload to the server with the api, thus updating the captions.
We need the following information for all the api's
| Heading | Description |
|---|---|
| record_id | Which recording are you trying to access |
| site | Address of the bigbluebutton server with your recording |
| checksum | every api request requires a checksum for authentication |
| bbb_secret | your bigbluebutton server secret you are trying to access |
| tts_secret | your tts application secret as the api endpoint is from tts |
| request | which request do you want (download_vtt/download_audio/upload_vtt) |
Below are some examples of how to send a request in ruby. Note we are using the JWT gem to encrypt the additional info. Also we are using the RestClient gem to send the request.
gem install "rest-client"
gem install "jwt"
Edit-Api usage:
remember to include the gems you are using
require "rest-client"
require "jwt"
download_vtt api
def download_vtt
puts "downloading vtt..."
record_id = 'your_record_id'
bbb_secret = 'bigbluebutton_secret'
request = "getRecordingTextTracksrecordID=#{record_id}#{bbb_secret}"
checksum = Digest::SHA1.hexdigest("#{request}")
site = 'https://my_bigbluebutton_server.ca' <--- example (put the address of your bigbluebutton server)
tts_secret = 'text_track_application_secret' <--- find in your credentials.yaml in tts
payload = { :bbb_url => site,
:bbb_checksum => checksum
}
token = JWT.encode payload, tts_secret, 'HS256' <--- JWT gem encoding information
request = RestClient::Request.new(
method: :post,
url: "https://localhost:4000/edit/downloadvtt/#{record_id}", <--- request being sent to tts (could be on localhost or a different server enter address appropriately)
payload: {:token => token}
)
response = request.execute
puts response
end
download_audio api
def download_audio
puts "downloading audio..."
record_id = 'your_record_id'
bbb_secret = 'bigbluebutton_secret'
request = "getRecordingsrecordID=#{record_id}"
request = request + bbb_secret
checksum = Digest::SHA1.hexdigest("#{request}")
site = 'https://my_bigbluebutton_server.ca' <--- example (put the address of your bigbluebutton server)
tts_secret = 'text_track_application_secret' <--- find in your credentials.yaml in tts
payload = { :bbb_url => site,
:bbb_checksum => checksum
}
token = JWT.encode payload, tts_secret, 'HS256' <--- JWT gem encoding information
request = RestClient::Request.new(
method: :post,
url: "https://localhost:4000/edit/downloadaudio/#{record_id}", <--- request being sent to tts (could be on localhost or a different server enter address appropriately)
payload: {:token => token}
)
response = request.execute
File.open("absolute_path/folder_where_you_want_to_save_audio_file/downloaded_audio.wav", 'wb') do |f|
f.write response.body
end
end
once you have edited your vtt(caption) file and are ready to put the new file on the server upload_vtt api
def upload_vtt
puts "uploading vtt..."
record_id = 'your_record_id'
site = 'https://my_bigbluebutton_server.ca' <--- example (put the address of your bigbluebutton server)
bbb_secret = 'bigbluebutton_secret'
kind = 'subtitles'
lang = 'en_US'
label = 'English'
request = "putRecordingTextTrackrecordID=#{record_id}&kind=#{kind}&lang=#{lang}&label=#{label}"
request += bbb_secret
checksum = Digest::SHA1.hexdigest(request)
tts_secret = 'text_track_application_secret' <--- find in your credentials.yaml in tts
payload = { :bbb_url => site,
:bbb_checksum => checksum,
:kind => kind,
:label => label,
:caption_locale => lang
}
token = JWT.encode payload, tts_secret, 'HS256' <--- JWT gem encoding information
request = RestClient::Request.new(
method: :post,
url: "https://localhost:4000/edit/uploadvtt/#{record_id}", <--- request being sent to tts (could be on localhost or a different server enter address appropriately)
payload: { :file => File.open('/folder_with_updated_vtt_file/captions_en-US.vtt', 'rb'), <--- make sure the vtt file you send is named captions_en-US.vtt
:token => token}
)
response = request.execute
end
Make sure the record_id being requested exist on the bbb server it is requested from. Make sure to name the parameters sent as shown in the table above.
To find bbb_secret on bbb server type bbb-conf --secret
To find tts_secret on tts server(could be same as bbb) type tts-secret or open credentials.yaml in the tts application
For the download audio make sure you provide a path to an existing folder so it can save the audio file. Also make sure the folder has correct permissions.
Make sure the path to the given vtt file to upload is correct. Also make sure the file has correct permissions. Make sure the file that you upload is named captions_en-US.vtt, if not the file will be uploaded but bbb will not display captions.
You will have to configure git on the server prior to this
git config --global user.email [email protected]
git config --global user.name example_username
The application has a deploy.sh file in the root. Edit it based on where the application path is on your server. Add this sh filer to your sudoers file as shown below
sudo visudo
After this just add the following line to the file and save and exit
texttrack ALL = NOPASSWD: /var/docker/text-track-service/deploy.sh
(in the given example texttrack is our tts user & we have entered the path the the deploy.sh in our application)
Now anytime you execute this deploy.sh file your application should update and restart itself
./deploy.sh