This guide is provided by LiveData. Agora is planning a documentation upgrade program for all extensions on the marketplace. Please stay tuned.
LiveData RTVT extension allows you to embed real-time voice transcription and translation into your mobile application providing automated speech recognition(ASR) without any upfront data training requirements.You can find the integration examples below.
The LiveData RTVT extension encapsulates the core APIs of the LiveData RTVT SDK. By calling the setExtensionProperty or setExtensionPropertyWithVendor method of the Agora Video SDK v4.x and passing in the corresponding key
and value
, you can quickly integrate capabilities of LiveData RTVT. For details, see the key-value overview.
The development environment has to meet the following requirements:
- Android
- Android Studio 4.1 or later
- A physical device (not an emulator) running Android 5.0 or later
- iOS
- Xcode 9.0 or later.
- A physical device (not an emulator) running iOS 11.0 or later.
The LiveData RTVT extension works together with the Video SDK v4.x. Refer to the following doc to integrate the SDK and start a basic voice call:
To receive a appKey
and a appSecret
from LiveData
- please buy and activate the extension project on Agora Console, then click View in the Secret column.
- or contact us via Agora.
- Download the Android package of LiveData RTVT from the Extensions Marketplace.
- Unzip the package, and save all
.aar
files to the/app/libs
path of your project folder. - In the
app/build.gradle
file, add the following line independencies
:
implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])
- Download the iOS package of LiveData RTVT from the Extensions Marketplace.
- Unzip the package, and save all
.framework
files to the<ProjectName>
path. - Ensure that you select Embed & Sign instead of Embed.
You can save .framework
files under your project folder, as follows:
.
├── <ProjectName>
├── <ProjectName>.xcodeproj
This section describes the call sequence you implement to use LiveData RTVT features in your app.
When you initialize AgoraRtcEngine
:
- first call
addExtension
to load the extension - then call
enableExtension
to enable the extension
RtcEngineConfig config = new RtcEngineConfig();
config.addExtension("agora-iLiveData-filter-pre");
config.addExtension("agora-iLiveData-filter-post");
engine = RtcEngine.create(config);
engine.enableExtension("iLiveDataPre", "RTVT_PRE", true);
engine.enableExtension("iLiveDataPost", "RTVT_POST", true);
When you initialize AgoraRtcEngine
, call enableExtensionWithVendor
to enable the extension.
AgoraRtcEngineConfig *config = [AgoraRtcEngineConfig new];
// set config.eventDelegate = self to accept the `onEvent` callback
// enable RTVT extension
[_agoraKit enableExtensionWithVendor:[iLiveDataSimpleFilterManager companyName]
extension:[iLiveDataSimpleFilterManager plugName]
enabled:YES];
Call setExtensionProperty
to specify key as startAudioTranslation_post
, startAudioTranslation_pre
and pass appkey
, appsecret
and other parameters in value in json format. appsecret` parameters in value.
JSONObject jsonObject = new JSONObject();
jsonObject.put("appKey", "");
jsonObject.put("appSecret", "");
jsonObject.put("srcLang", "zh");
jsonObject.put("dstLang", "en");
jsonObject.addProperty("asrResult", true);
jsonObject.addProperty("transResult", true);
jsonObject.addProperty("tempResult", false);
ExtensionInfo extensionInfo = new ExtensionInfo();
extensionInfo.localUid = ;
extensionInfo.channelId = ;
extensionInfo.remoteUid = ;
-Post
engine.setExtensionProperty("iLiveDataPost", "RTVT_POST", extensionInfo, "startAudioTranslation_post", jsonObject.toString());
-Pre
engine.setExtensionProperty("iLiveDataPre", "RTVT_PRE", "startAudioTranslation_pre", jsonObject.toString());
When you are ready to start using RTVT, call setExtensionPropertyWithVendor
and pass in the corresponding keys and values:
- set key as
startAudioTranslation
- set value as
appkey
,appsecret
,srclang
dstLang
in JSON
NSDictionary * translateDic = @{
@"appKey":@"appKey",
@"appSecret":@"appSecret",
@"srcLanguage":@"zh",
@"destLanguage":@"en"
};
NSData * translateDicJsonData = [NSJSONSerialization dataWithJSONObject:translateDic options:NSJSONWritingPrettyPrinted error:nil];
NSString * translateDicJsonString = [[NSString alloc] initWithData:translateDicJsonData encoding:NSUTF8StringEncoding];
[_agoraKit setExtensionPropertyWithVendor:[iLiveDataSimpleFilterManager companyName]
extension:[iLiveDataSimpleFilterManager rtvt_plugName])
key:@"startAudioTranslation"
value:translateDicJsonString];
When stop using RTVT, call setExtensionProperty
and pass in the corresponding keys and values:
engine.setExtensionProperty("iLiveDataPost", "RTVT_POST", "closeAudioTranslation_post", "{}");
- 前处理
engine.setExtensionProperty("iLiveDataPre", "RTVT_PRE", "closeAudioTranslation_pre", "{}");
When stop using RTVT, call setExtensionPropertyWithVendor
and pass in the corresponding keys and values:
- set key as
closeAudioTranslation
- set value as
end
[_agoraKit setExtensionPropertyWithVendor:[iLiveDataSimpleFilterManager companyName]
extension:[iLiveDataSimpleFilterManager rtvt_plugName]
key:"closeAudioTranslation"
value:"end"];
LiveData RTVT extension provides result of voice transcription and translation, you can receive all return of results or either of them, using onEvent
callback of the Agora SDK.
After enable LiveData RTVT extension, you can receive the results of voice transcription and translation via onEvent
callback. The description of onEvent
callback keys can be seen below.
- Post
@Override
public void onEvent(String vendor, String extension, String key, String value) {
vendor: “iLiveDataPost”
key: “recognizeResult ‘RecognizeResult identifier ’translateResult ‘TranslateResult ’TranslateResult identifier ‘recognizedTempResult ’Temporary RecognizeResult identifier ‘translatedTempResult ’TemporaryTranslateResult identifier
extension: “RTVT_POST”
value: the corresponding keys are recognized result and translated result and temporary recognized result and temporary translated result, respectively.
}
- Pre
@Override
public void onEvent(String vendor, String extension, String key, String value) {
vendor: “iLiveDataPre”
key: “recognizeResult ‘RecognizeResult identifier ’translateResult ‘TranslateResult ’TranslateResult identifier ‘recognizedTempResult ’TemporaryResult identifier ‘translatedTempResult ’TemporaryTranslateResult identifier
extension: “RTVT_PRE”
value: the corresponding keys are recognized result and translated result and temporary recognized result and temporary translated result, respectively.
}
After enable LiveData RTVT extension, you can receive the results of voice transcription and translation via onEvent
callback. The description of onEvent
callback keys can be seen below.
-(void)onEvent:(NSString *)provider extension:(NSString *)extension key:(NSString *)key value:(NSString *)value{
provider:"iLiveData"
extension:"RTVT"
key: "recognizeResult" "translateResult"
value:
{
""result",
"startTs",
"endTs",
}
}
The complete sample code and project is provided on GitHub:
Platform | Language | Sample Project |
---|---|---|
Android | Java | Sample Project |
iOS | Objective-C | Sample Project |
- git clone:
git clone https://github.com/highras/rtvt-agora-marketplace.git
- Open the Sample project in Android Studio.
- Gradle sync with the project.
- Open
app/src/main/res/values/string_configs.xml
file:- change
agora_app_id
andagora_access_token
to your own Agora project information. - change
livedata_translate_pid
andlivedata_translate_key
to your own LiveData RTVT extension information。
- change
- Using a physical Android device (not an emulator) to run the project.
- git clone:
git clone https://github.com/highras/rtvt-agora-marketplace.git
- Open the project in Xcode. Access to the
iOS/SW_Test
path, and run the CocoaPods command as below:
pod install
- Open the sample project
SW_Test.xcworkspace
in Xcode. - Open
iOS/SW_Test/SW_Test/ViewController.mm
file:- Fill in you own project information
appId
andToken
from Agora console. - Fill in your own LiveData RTVT extension information
appKeyRTVT
andappSecretRTVT
from Agora console。
- Fill in you own project information
- Using a physical iOS device (not an emulator) to run the project.
After running the project successfully, LiveData RTVT sample project will be installed on you Android or iOS device.
- Start the sample, fill in the channel number in the input box and click
Join
. The buttonJoin
will be change toLeave
after join the channel successfully. - Click the button
start translation
, speak to the device microphone, and you will see the real-time transcription and translation on the screen. - Click the button
stop translation
to stop transcription and translation. - Click the button
stop
to stop running LiveData RTVT sample project.
This section lists the APIs related to using extensions with the Agora SDK.
- addExtension in the
RtcEngineConfig
class - enableExtension in the
RtcEngine
class - setExtensionProperty in the
RtcEngine
class - getExtensionProperty in the
RtcEngine
class - onEvent in the
IMediaExtensionObserver
class
- enableExtensionWithVendor in the
AgoraRtcEngineKit
class - setExtensionPropertyWithVendor in the
AgoraRtcEngineKit
class - getExtensionPropertyWithVendor in the
AgoraRtcEngineKit
class - onEvent in the
AgoraMediaFilterEventDelegate
class
To implement the LiveData RTVT extension in app, you need to pass in the corresponding key-value pair。
When calling setExtensionProperty
or setExtensionPropertyWithVendor
, you can pass keys:
Key | Description |
---|---|
startAudioTranslation | start real-time voice transcription and translation |
closeAudioTranslation | stop real-time voice transcription and translation |
The Agora SDK onEvent
contain keys:
Key | Description |
---|---|
recognizeResult | result of real-time voice transcription |
translateResult | result of real-time voice translation |
start | error message when calling startAudioTranslation |
Value | Description |
---|---|
appKey | buy and activate the extension project on Agora Console, then click View in the Secret column |
appSecret | buy and activate the extension project on Agora Console, then click View in the Secret column. |
srcLanguage | source language, see language supported list |
destLanguage | destination language, see language supported list |
Value | Description |
---|---|
end | stop real-time voice transcription and translation |
Value | Description |
---|---|
{"result", "startTs", "endTs"} | result of real-time voice transcription |
Value | Description |
---|---|
{"result", "startTs", "endTs"} | result of real-time voice translation |
Value | Description |
---|---|
start | error message when calling startAudioTranslation |