Using ADBKeyboard for injecting text #1751

excitoon · 2020-09-10T13:51:42Z

No description provided.

rom1v

Thank you for the PR 👍

Ideally, I would like the IME code to be included in scrcpy-server: #37 (comment)

Moreover, this would also allow a socket connection between the server and the IME (to minimize latency).

However, until this is implemented (I have zero time to work on this currently), I think calling ADBkeyboard is an acceptable solution.

If this option is passed, I think it should also save the current IME, set the ADBKeyboard IME, and it should restore the previous one on cleanup.

rom1v · 2020-09-10T14:34:20Z

server/src/main/java/com/genymobile/scrcpy/Controller.java

+ // Process latin keys the same way in order to provide same reaction speed.
+ try {
+ Process process = Runtime.getRuntime().exec("am broadcast -a ADB_INPUT_CHARS --eia chars " + String.valueOf((int) c));
+ return process.waitFor() == 0;


You should send the broadcast programmatically (to avoid execute a new process and add latency for every letter).

I guess this involves retrieving an IActivityManager by calling ActivityManager.getService() (by introspection) (or ActivityManagerNative.getService() on older versions), and calling broadcastIntent().

Is that method stable enough? I only have A9 devices and there is something made about activities in A10. I'm not Android developer though.

rom1v · 2020-09-10T14:37:39Z

app/src/cli.c

@@ -651,6 +651,7 @@ guess_record_format(const char *filename) {
 #define OPT_DISABLE_SCREENSAVER 1020
 #define OPT_SHORTCUT_MOD 1021
 #define OPT_NO_KEY_REPEAT 1022
+#define OPT_USE_ADB_KEYBOARD 1026


Some numbers don't exist

Thank you for the PR +1

Ideally, I would like the IME code to be included in scrcpy-server: #37 (comment)

Moreover, this would also allow a socket connection between the server and the IME (to minimize latency).

However, until this is implemented (I have zero time to work on this currently), I think calling ADBkeyboard is an acceptable solution.

If this option is passed, I think it should also save the current IME, set the ADBKeyboard IME, and it should restore the previous one on cleanup.

All of these is right. 👍

Artur202030 · 2020-09-15T12:14:20Z

https://github.com/Lurker00/scrcpy/releases/tag/v1.10L0
The new features are:
-10-fingers multitouch support.
-Any language input support with an aid of ADBKeyBoard (included).
...

rom1v · 2020-11-09T20:47:26Z

Hi,

I just implemented what I suggested here, and now it works great (without latency).

You could test my work-in-progress branch adbkeyboard.

scrcpy --use-adb-keyboard

What remains to do:

automatically select the ADB keyboard on start
automatically restore the initial keyboard on stop (even if that won't work if we start 2 instances of scrcpy with ADBKeyboard, since the second one will consider that the initial keyboard was ADBKeyboard).
some cleanup

excitoon · 2020-11-09T20:50:56Z

Hey. You can just never restore keyboard to ADBKeyboard, that shall solve the problem. Kind regards, Vladimir.

…

On Mon, Nov 9, 2020, 11:47 PM Romain Vimont ***@***.***> wrote: Hi, I just implemented what I suggested here <#1751 (comment)>, and now it works great (without latency). You could test my work-in-progress branch adbkeyboard <https://github.com/Genymobile/scrcpy/commits/adbkeyboard>. scrcpy --use-adb-keyboard What remains to do: - automatically select the ADB keyboard on start - automatically restore the initial keyboard on stop (even if that won't work if we start 2 instances of scrcpy with ADBKeyboard, since the second one will consider that the initial keyboard was ADBKeyboard). - some cleanup — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1751 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARGSB6AQO3LGR3TBX7DUDTSPBIO3ANCNFSM4RFC3LEQ> .

rom1v · 2020-11-09T21:18:55Z

A future next step could be to include the IME in scrcpy (as a real android app), to avoid the requirement to install a separate app.

I'm wondering if it's still possible to create a real/installable android app without gradle (using build-tools directly). The reason is that the official debian package uses build_without_gradle.sh because the full Android SDK is not available in Debian official repositories.

Artur202030 · 2020-11-10T00:52:25Z

hi,
both English and Russian letters are printed simultaneously.

rom1v · 2020-11-10T06:10:53Z

@Artur202030 Try with:

scrcpy --use-adb-keyboard --prefer-text

rom1v · 2020-11-10T08:01:50Z

An issue with using the existing ADBKeyBoard is the absence of permissions.

As a consequence, once enabled, any installed app could inject any text or events (anything managed by AdbIME) as they like. This is a security issue IMO.

To avoid the problem, the IME should declare a permission, and with shell rights we can send a broadcast with this permission. In short, the equivalent of (not tested):

adb shell am broadcast -a com.com.myexample.MY_ACTION --receiver-permission com.myexample.permission.MY_PERMISSION

Note that this may only work for broadcasts, not services or activities. For these cases (or if the solution above does not work), we could require some shell permission. But it will cause issues for XiaoMI devices: Genymobile/gnirehtet#5 (and gnirehtet may not be started by broadcast receivers Genymobile/gnirehtet@f868180).

Artur202030 · 2020-11-10T19:24:59Z

@ Artur202030 Попробуйте:
scrcpy --use-adb-keyboard --prefer-text

Sorry your code works without delay

rom1v · 2020-11-10T19:44:23Z

so normal but 1 sec delay

Which branch?

Artur202030 · 2020-11-10T20:12:28Z

i tested your code and excitoon. Mixed up files.
Sorry.

rom1v · 2020-11-10T20:25:56Z

Could you confirm that there is no delay on my branch (#1751 (comment))?

Artur202030 · 2020-11-10T20:49:55Z

Could you confirm that there is no delay on my branch (#1751 (comment))?

Your code works without delay.
Thanks.

quyleanh · 2020-11-30T17:09:24Z

@excitoon could you please take a look at this PR again?

quyleanh · 2020-12-01T08:59:05Z

@rom1v I have built my self adbkeyboard and used scrcpy --use-adb-keyboard, but I still cannot enter Vietnamese character.

Is there any way to report log?

quyleanh · 2020-12-14T16:50:48Z

@rom1v could you please take a look at this PR?

rom1v · 2020-12-14T17:19:02Z

@quyleanh I implemented broadcasts so that it works without latency (#1751 (comment)), but the absence of permissions of AdbKeyboard is a problem (#1751 (comment)). Also, scrcpy should enable/disable the keyboard on start/stop.

I consider adding a feature to "install" the scrcpy server instead. That way, it could provide its own IME: #1880 (comment)

but I still cannot enter Vietnamese character.

It should allow to inject any UTF-8 chars (try to copy paste a vietnamese character with Alt+Shift+v to inject characters).

quyleanh · 2020-12-15T02:32:18Z

@rom1v thank you for your kind response.

It should allow to inject any UTF-8 chars (try to copy paste a vietnamese character with Alt+Shift+v to inject characters).

I know that copy paste feature. But I would like to inject UTF-8 characters directly from keyboard.

@quyleanh I implemented broadcasts so that it works without latency (#1751 (comment)), but the absence of permissions of AdbKeyboard is a problem (#1751 (comment)). Also, scrcpy should enable/disable the keyboard on start/stop.

I consider adding a feature to "install" the scrcpy server instead. That way, it could provide its own IME: #1880 (comment)

So in summary, we will not have inject UTF-8 character directly from keyboard soon? As I understand, there are a lot of problem needed to do.

rom1v · 2020-12-15T08:38:20Z

I know that copy paste feature. But I would like to inject UTF-8 characters directly from keyboard.

Yes, I know, but since you said that Vietnamese characters were not injected, I wanted to check where it failed.

There are two copy-paste:

once using the clipboard, where any byte stream work (including UTF-8)
once injecting the characters, which acts as if every character was pressed from a keyboard (injected char by char), currently it only supports ASCII (+ some other accented chars)

If you use the second method with the ADBKeyboard, I expect that Vietnamese characters work. Is it the case?

quyleanh · 2020-12-15T09:05:11Z

If you use the second method with the ADBKeyboard, I expect that Vietnamese characters work. Is it the case?

Yes, I use the second method. There are only some Vietnamese characters work.
For example, if I type ô (unicode hex is U+00F4), it is fine. But if I type ồ (unicode hex is U+1ED3), the warning appreases.
[server] WARN: Could not inject char u+1ed3

You can refer to this summary table.

rom1v · 2020-12-15T09:08:23Z

[server] WARN: Could not inject char u+1ed3

You're not running my branch adbkeyboard with scrcpy --use-adb-keyboard: #1751 (comment)

quyleanh · 2020-12-15T09:17:24Z

You're not running my branch adbkeyboard with scrcpy --use-adb-keyboard: #1751 (comment)

I build your branch and run it now, but I forgot to use scrcpy --use-adb-keyboard syntax.
I checked again with scrcpy --use-adb-keyboard, but I cannot enter any Vietnamese characters now.
How can I provide log since there is no output or warning on terminal?

rom1v · 2020-12-15T09:20:42Z

Try with AdbKeyboard alone: https://github.com/senzhk/ADBKeyBoard (without scrcpy).

quyleanh · 2020-12-15T09:27:55Z

Thank you for your suggestion but I want to use scrcpy for both mirroring screen and injecting text.
There is no way after all?

rom1v · 2020-12-15T09:29:42Z

Yes, I got it, but try with AdbKeyboard alone to check if Vietnamese characters work. If they don't, then it's expected that the adbkeyboard branch can't do it either. If it works, then there is a problem somewhere else.

quyleanh · 2020-12-15T10:02:58Z

Thank you for your detail instruction. The senzhk/ADBKeyBoard works well. This is my test:

adb shell am broadcast -a ADB_INPUT_TEXT --es msg 'Hôm nay trời rất là đẹp. Ồ ngoài kia có chú ngựa ô. Ổ gà. Ở nhà không đi học. Hỏi han. Hello?'
Broadcasting: Intent { act=ADB_INPUT_TEXT flg=0x400000 (has extras) }
Broadcast completed: result=0

Is there any other suggestion?

rom1v · 2020-12-15T10:05:52Z

Test with ints:

adb shell am broadcast -a ADB_INPUT_CHARS --eia chars 7891

(7891 is 0x1ed3)

quyleanh · 2020-12-15T10:11:45Z

It's fine. The result is ồ.

rom1v · 2020-12-15T11:42:37Z

OK so in theory it should work with scrcpy on adbkeyboard. I don't know why it does not, but will not investigate for now, since in the end scrcpy should have its own IME (#1751 (comment)).

quyleanh · 2020-12-16T02:13:57Z

Actually I have to use an external IME to type and inject Vietnamese on Windows. However other apps are OK with it.
In the previous test, all Vietnamese characters are injected from terminal since I already paste it in adb command. I don't know is there any different between injecting directly from adb command and injecting from other IME.

By the way, I'm glad to hear that you are working on a work around. Could you please push it in this repo, so I can try it? Or if it needs to test, just let me know.

Helaer · 2021-10-31T11:28:40Z

@rom1v Using ADBKeyboard to input text is a good solution, which can solve the problem that Samsung and other devices cannot paste text, Why is there no progress?

yilksd · 2022-01-10T09:27:01Z

I'm using ubuntu 20.04 + fcitx, and I have found 2 problems when using adbkeyboard:
1.adbkeyboard won't stop working when scrcpy window loses focus and I type in a QT application , so the characters also appear in scrcpy . The characters won't appear in scrcpy when I type in Firefox.
2.It seems there is a limitation in string's length. If I type a long sentence , only the first several characters can be injected. There is no such a limitation in adbkeyboard's ADB_INPUT_CHARS feature because I can inject such a long sentence using am broadcast -a ADB_INPUT_CHARS --eia chars 'xxx' , so maybe the limitation is in another place.

dieskim · 2024-09-09T05:09:07Z

@rom1v - I was wondering if there is any update on "scrcpy having its its own IME" ? We tried scrcpy on our current stack and even with "--keyboard=uhid" we are still having clipboard paste issue on Samsung devices etc. Switching keyboard inputs on the device is also needed when switching between languages. In the past we have used Vysor with its own IME and that has been a good solution for different language inputs and clipboard management.

We are happy to put in work to make this happen if thats something that could be helpful? Maybe getting --use-adb-keyboard into the main release could be one way? Or a custom scrcpy IME apk that can be installed?

Thanks for all the work you do.

rom1v · 2024-09-09T09:53:45Z

I was wondering if there is any update on "scrcpy having its its own IME" ?

No, since most of the need is now covered by UHID keyboard.

I understand that there are still issues that UHID keyboard does not solve, but I am not keen on adding a new subproject just for that.

Installing AdbKeyboard can also be a problem for security: any app on the device could inject text (#1751 (comment)).

dieskim · 2024-09-09T11:59:23Z

Yes I understand. Thanks for getting back to me. I wonder if adding a cli option that would enable a “keyboard broadcaster”, or something similar, from scrcpy would be something that you might be willing to add to the core? Looking at how adbkeyboard was implemented before in the adbkeyboard branch, we could capture the keyboard inputs and broadcast those, we could even add a secret key for security that scrcpy could broadcast with the broadcast and then a custom IME app could listen for those broadcasts. Not sure if we could add to adbkeyboard but making something similar should be possible and most likely not too hard. Let me know what you think, we could look into it more. If it’s something you would be willing to ado into the core if done correctly we could most likely put in the work to get it done. We have looked into using scrcpy in our development and testing stack before but different language inputs and clipboard management has always been an issue for us.

…

On Mon, Sep 9, 2024 at 16:54 Romain Vimont ***@***.***> wrote: I was wondering if there is any update on "scrcpy having its its own IME" ? No, since most of the need is now covered by UHID keyboard. I understand that there are still issues that UHID keyboard does not solve, but I am not keen on adding a new subproject just for that. Installing AdbKeyboard can also be a problem for security: any app on the device could inject text (#1751 (comment) <#1751 (comment)>). — Reply to this email directly, view it on GitHub <#1751 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJIUKMTEJ2CVN6LAP6UKFLZVVV4DAVCNFSM6AAAAABN3W7Z6GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZXGY3DCNRWGI> . You are receiving this because you commented.Message ID: ***@***.***>

Using ADBKeyboard for injecting text.

694cad3

rom1v reviewed Sep 10, 2020

View reviewed changes

rom1v mentioned this pull request Nov 19, 2020

(1.16) (Linux) the "--prefer-text" parameter does not work #1920

Open

rom1v mentioned this pull request Nov 20, 2021

Inject UTF-8 text #1426

Closed

CyberShadow mentioned this pull request Jan 7, 2022

Pull master into adbkeyboard #2904

Open

kcpr mentioned this pull request Dec 5, 2022

Modifier keys not "fully" forwarded (not detected by Keymapper)? #3609

Open

2 tasks

Using ADBKeyboard for injecting text #1751

Are you sure you want to change the base?

Using ADBKeyboard for injecting text #1751

Conversation

excitoon commented Sep 10, 2020

rom1v left a comment

Choose a reason for hiding this comment

rom1v Sep 10, 2020

Choose a reason for hiding this comment

excitoon Sep 10, 2020

Choose a reason for hiding this comment

rom1v Sep 10, 2020

Choose a reason for hiding this comment

excitoon Sep 10, 2020

Choose a reason for hiding this comment

Artur202030 commented Sep 15, 2020

rom1v commented Nov 9, 2020

excitoon commented Nov 9, 2020 via email

rom1v commented Nov 9, 2020

Artur202030 commented Nov 10, 2020

rom1v commented Nov 10, 2020

rom1v commented Nov 10, 2020

Artur202030 commented Nov 10, 2020 • edited Loading

rom1v commented Nov 10, 2020

Artur202030 commented Nov 10, 2020

rom1v commented Nov 10, 2020

Artur202030 commented Nov 10, 2020

quyleanh commented Nov 30, 2020

quyleanh commented Dec 1, 2020 • edited Loading

quyleanh commented Dec 14, 2020

rom1v commented Dec 14, 2020

quyleanh commented Dec 15, 2020

rom1v commented Dec 15, 2020

quyleanh commented Dec 15, 2020

rom1v commented Dec 15, 2020

quyleanh commented Dec 15, 2020

rom1v commented Dec 15, 2020

quyleanh commented Dec 15, 2020

rom1v commented Dec 15, 2020

quyleanh commented Dec 15, 2020

rom1v commented Dec 15, 2020

quyleanh commented Dec 15, 2020

rom1v commented Dec 15, 2020

quyleanh commented Dec 16, 2020

Helaer commented Oct 31, 2021

yilksd commented Jan 10, 2022

dieskim commented Sep 9, 2024 • edited Loading

rom1v commented Sep 9, 2024

dieskim commented Sep 9, 2024 via email

Artur202030 commented Nov 10, 2020 •

edited

Loading

quyleanh commented Dec 1, 2020 •

edited

Loading

dieskim commented Sep 9, 2024 •

edited

Loading