Add Digital PTZ via mouse and touch #491

dbuezas · 2023-05-21T21:34:44Z

Adds configurable Digital (local, fake, css-based) zoom and panning via mouse and touch.
The viewport setup can be persisted so it survives page reloads, and seamlessly adapts between landscape, portrait and full screen modes.

The navigation gestures are the same as in google maps and can be disabled one by one.

For you to consider:

~~I enabled all gestures by default.~~
~~This may make scrolling difficult for people that has multiple videos in the same panel.~~
~~It may be more reasonable to enable only these by default~~

digital_ptz:
  mouse_drag_pan: true
  mouse_wheel_zoom: true
  mouse_double_click_zoom: true

  touch_pinch_zoom: true
  touch_drag_pan: true
  touch_tap_drag_zoom: true

  persist: true

Persistence is enabled by default
I am not sure if there is any setting fro WebRTC with which all of this would be incompatible. Please let me know.
It ended up being a fair bit of code, partially because I made extra effort in dividing into multiple classes for easier maintenance.
To avoid silly mistakes, I actually made this in typescript and transpiled it back to js.

I'm quite happy with how it feels and find it quite useful myself. I'm looking forward to your feedback :)

dbuezas · 2023-05-22T07:32:03Z

Some combinations of options behave unexpectedly. I'll make a second pass tonight.

dbuezas · 2023-05-23T20:14:19Z

I've been testing it for a couple of days and have polished a couple of rough edges.

dbuezas · 2023-05-24T18:28:31Z

I think I'm satisfied now. I made a (hopefully) last round of cleanup, simplifications and optimisations.
Looking forward to your feedback, I'm using it quite a bit myself and it feels very natural to me. I think if feels as if it was natively supported by the video tag.

zelo66 · 2023-05-31T13:18:47Z

Great work,

1.I've been testing this for a week and it works great on my phone and in regular preview on my computer. Unfortunately in full screen mode on PC it doesn't work for me. I tested on two computers, in chrome and opera.

My settings:

digital_ptz:
  mouse_drag_pan: true
  mouse_wheel_zoom: true
  mouse_double_click_zoom: true
  touch_pinch_zoom: true
  touch_drag_pan: false
  touch_tap_drag_zoom: true
  persist: false

can I use the buttons in the UI to zoom in and out?
Such as you have given in your example with lovelace-card-mod
can I add a button that will point and zoom in the right way to the desired location? E.g. I will set a gate position and when I press the gate button the card will automatically zoom in and move to the selected location?

dbuezas · 2023-06-01T21:12:06Z

Hey @zelo66,
thanks for the feedback.

For it to work in full screen in desktop, you need to enable webrtc's custom ui

ui: true

There's no support for zooming with buttons at the moment. I could add it as a new feature after this PR is merged (if Alexey agrees)
Also no feature available to set and activate presets. At some point I want to add an option to draw a box to zoom there. If/when I do, I can add some connection to trigger that via buttons.

dbuezas · 2023-06-03T20:30:57Z

Hey @AlexxIT I assume you just haven't had time yet, but let me know if there is something about the code you'd like to be implemented differently (e.g changing defaults, explaining something with comments, ...)

I understand that it is not a very small PR, I just can't find a way to make this more compact without compromising readability or corner cases (like handling full screen video).

zelo66 · 2023-06-04T16:08:50Z

Hey @zelo66, thanks for the feedback.

For it to work in full screen in desktop, you need to enable webrtc's custom ui
ui: true
There's no support for zooming with buttons at the moment. I could add it as a new feature after this PR is merged (if Alexey agrees)

Also no feature available to set and activate presets. At some point I want to add an option to draw a box to zoom there. If/when I do, I can add some connection to trigger that via buttons.

Of course, I have set
ui:true

By it not working I meant that the image zooms in just a little. I have recorded a gif. On my main camera the image zooms in even less (there is a zoom effect which disappears after about 0.5 seconds.

I checked chrome, opera and firefox.

In Chrome and Opera zooming in full screen has the effect as in the gif. In Firefox the zoom works very well, but the " mouse_double_click_zoom" completely does not work.

dbuezas · 2023-06-04T16:14:24Z

Do you have any custom styles by any chance?

zelo66 · 2023-06-04T16:41:58Z

No, everything is in the default settings.

I have now checked in one more instance of HA, plus on another computer. Same effect. It zooms in a bit, looks like the aspect ratio changes and blocks further zooming.

dbuezas · 2023-06-04T20:17:40Z

I can't reproduce this so I am not sure what could be happening. Please share the yaml of the card and go2rtc stream you are using for the example. Also version of HA and OS of the pc you recorded the gif with.
Also please make sure you are running the latest commit of this PR

zelo66 · 2023-06-08T18:42:35Z

type: custom:webrtc-camera
url: rtsp:https://zephyr.rtsp.stream/movie?streamKey=19b175b07ab0c45c0d22c3fa68797415
poster: https://home-assistant.io/images/cast/splash.png
muted: true
ui: true
webrtc: true
digital_ptz:
  mouse_drag_pan: true
  mouse_wheel_zoom: true
  mouse_double_click_zoom: true
  touch_pinch_zoom: true
  touch_drag_pan: false
  touch_tap_drag_zoom: true
  persist: false

For this example, I am using a test stream from the website:
https://rtsp.stream/

HA:
Home Assistant 2023.5.4 Supervisor 2023.06.1 Operating System 10.2 Interfejs użytkownika: 20230503.3 - latest

OS:
Win 10 Home 22H2

Of course I use the latest commit. I've also found that when I turn on the developer options the zoom works normally in full screen mode. (As long as developer mode is open).

dbuezas · 2023-06-09T16:59:34Z

It's probably windows related. I don't have a windows machine to debug it. Do you think it is a blocker for merging?

AlexxIT · 2023-06-16T08:38:48Z

Hi @dbuezas. Just had time to look at your work. Two things bother me:

A lot of extra external files for just one feature that only a part of users will use. Each will be downloaded to the client individually. Not the best approach in modern web development. Perhaps one file would be enough. Or even including all the code in webrtc-camera.js. Because video-rtc.js is the streaming core from go2rtc and is copied "as is". And webrtc-camera.js has all the extra functionality.
I'm not sure that all constructions will work in all browsers. Especially in horrible iOS older versions. Users of old iOS are also human beings and deserve to have normal streaming cameras. Of course, as far as Apple has allowed it. I'm not sure about constructions like ...

dbuezas · 2023-06-16T20:20:29Z

Hi @AlexxIT,

1. Multiple files

I think the browser will reuse the same http connection to grab all files efficiently. I read it is only discouraged if the dependency tree is very deep, and I am afraid my code will make the webrtc-camera.js too big to work comfortably with. But anyway, your repo your rules :)

What do you prefer?

One dedicated file with all the functionality
All inside the webrtc-camera.js file
A small and simple bundling script so you can modularise everything but serve a single file (if you want i can use parceljs which requires no extra config files)

2. Modern javascript syntax

I checked the browser versions required for each modern operator:

import webrtc-camera.js already uses import so the lowest supported version is Safari 11.6
=> Safari 10
class Safari 9
`const Safari 11
... Safari 8
`ES6 ${template} literals` Safari 9.1

So it looks like we are safe. Again, your repo, so if you just don't like a particular operator I can remove it.

Let me know and I'll apply your request on the weekend.
If you don't actually like the feature yourself, I can of course just make a fork for the screen pinchers like me, I'll understand that too.

AlexxIT · 2023-06-17T03:47:09Z

I think one extra file can have all zoom functionality
Looks like my fears were wrong

…ions, made touch_drag_pan default to false

… zoom

…e aspect ratio of the video is important

Also, don't prevent scrolling on touch drag unless zoom scale != 1

dbuezas · 2023-06-18T09:55:44Z

Hey @AlexxIT,
as requested, all the code is now in a single file digital-ptz.js.
I also:

Rebased so it can be merged.
Fixed interference between touch panning and scrolling in mobile (scroll unless zoomed), which now allows for enabling it by default
Added a way to disable digital ptz completely via digital_ptz: false

Let me know if you find something else you'd like to be done differently. I obviously like the feature so I enabled it by default, but you may prefer it to be disabled by default instead and require writing digital_ptz: to enable it

dbuezas · 2023-07-06T12:08:35Z

Hey Alex!
Does anything else need clarification or improvement?

AlexxIT · 2023-07-06T16:01:58Z

No. I'm planning to include this in next release. Just needs some time.

AlexxIT · 2023-07-06T18:21:46Z

Just as I thought - the iOS12 test did not pass.

I know it's a very old browser. But Hass somehow works in it. And WebRTC somehow works in it.

And with these innovations, the whole WebRTC card is falling down :(

PS. What you think about external "HACS Frontend plugin" for WebRTC card? Maybe it's possible to add plugins functionality to WebRTC card.

dbuezas · 2023-07-06T18:28:48Z

Oh, no!
Do you know exactly what failed? I can try to fix it quickly

AlexxIT · 2023-07-06T18:46:29Z

I have left many comments to code

dbuezas · 2023-07-06T18:55:04Z

I don't see them yet. Maybe is the review not marked as finished?

custom_components/webrtc/www/digital-ptz.js

dbuezas · 2023-07-06T19:54:03Z

Ok, it should work now.
I ensured the code is compatibly by finding out what ECMAScript version is compatible with iOS12 (it is ES2015) and setting the typescript transpilation target to it.

Sorry for that

custom_components/webrtc/www/webrtc-camera.js

…eatures

dbuezas · 2023-07-09T10:36:08Z

I applied all fixes and moved initialisation to its own member function so it is consistent with the rest of your code.

…ntrol

AlexxIT · 2023-07-10T04:40:55Z

Looks like all works fine now. But pinch action enables full screen on iPhone. But it is not your problem :)

AlexxIT · 2023-07-10T04:44:48Z

Thanks to you, I have researched ESLinter and will add it to the next release. It will help to avoid such compatibility problems.

dbuezas · 2023-07-10T04:47:50Z

That's a very good idea!
And prettier is great for code formatting. If you haven't tried typescript yet, I think you'd like it too :)

AlexxIT · 2023-07-10T06:37:11Z

I don't want to implement any JS build stage to my projects yet. It will unnecessary increase complexity.

PS. I'm using prettier from PyCharm.

dbuezas · 2023-07-10T06:48:27Z

True, it does add a step. I mostly work with typescript so I'm very used to it.
For digital-ptz.js I used prettier locally, let me know if I should use a different format

dbuezas · 2023-07-13T05:56:21Z

I didn't consider people using style transforms themselves. I'll try adding a wrapper element around the video and transform that one instead this way they should be able to coexist.

AlexxIT · 2023-07-13T09:32:26Z

They use it only for rotation and changing aspect ratio

dbuezas mentioned this pull request May 21, 2023

Digital zoom controls (via css) #482

Closed

dbuezas marked this pull request as draft May 22, 2023 07:31

dbuezas marked this pull request as ready for review May 22, 2023 21:14

dbuezas mentioned this pull request May 29, 2023

Zoom video stream (not via PTZ) #438

Closed

dbuezas added 11 commits June 18, 2023 11:06

Add Digital PTZ via mouse and touch

4a3469e

Correct file name

2b41e18

Fix double-tap-to-zoom when touch_pan is disabled

cfe175c

Made gestures independent of each other for simpler code, renamed opt…

0c16f3d

…ions, made touch_drag_pan default to false

undo unintended linting to __init__.py

4b6e584

rename files to avoid dots

ee2a412

Reduce didMove threshold

63d4817

Default to no touch_tap_drag_zoom

95b5607

Fix NaN coordinates bug when reopening a video panel

992341b

Update defaults in readme

686f5c6

Optimize gestures, reenable tap-drag-zoom, transition on double click…

f626947

… zoom

Remove 3 unused instance variables and added an explanation of why th…

15147f4

…e aspect ratio of the video is important

dbuezas force-pushed the Add-digital-ptz-mouse-and-touch-control branch from 49cac43 to d0829c0 Compare June 18, 2023 09:10

Merge all digital-ptz code into a single file

1e15ec0

Also, don't prevent scrolling on touch drag unless zoom scale != 1

dbuezas force-pushed the Add-digital-ptz-mouse-and-touch-control branch from d0829c0 to 1e15ec0 Compare June 18, 2023 09:41

dbuezas added 3 commits June 18, 2023 11:43

Enable touch drag pan by default now that it only acts if scale != 1

4800e50

Add option to disable completely and update readme

644d611

Undo unintended py formatting

c53e9d6

AlexxIT reviewed Jul 6, 2023

View reviewed changes

dbuezas commented Jul 6, 2023

View reviewed changes

custom_components/webrtc/www/webrtc-camera.js Outdated Show resolved Hide resolved

iOS12 compatibility

ed5450c

dbuezas force-pushed the Add-digital-ptz-mouse-and-touch-control branch from 503641b to ed5450c Compare July 6, 2023 21:49

Move digitalPTZ init code to its own member function like all other f…

e3b14dd

…eatures

Merge branch 'AlexxIT:master' into Add-digital-ptz-mouse-and-touch-co…

9e62934

…ntrol

AlexxIT merged commit ee3600f into AlexxIT:master Jul 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Digital PTZ via mouse and touch #491

Add Digital PTZ via mouse and touch #491

dbuezas commented May 21, 2023 •

edited

Loading

dbuezas commented May 22, 2023

dbuezas commented May 23, 2023

dbuezas commented May 24, 2023

zelo66 commented May 31, 2023 •

edited by AlexxIT

Loading

dbuezas commented Jun 1, 2023

dbuezas commented Jun 3, 2023

zelo66 commented Jun 4, 2023

dbuezas commented Jun 4, 2023

zelo66 commented Jun 4, 2023

dbuezas commented Jun 4, 2023 •

edited

Loading

zelo66 commented Jun 8, 2023

dbuezas commented Jun 9, 2023

AlexxIT commented Jun 16, 2023 •

edited

Loading

dbuezas commented Jun 16, 2023

AlexxIT commented Jun 17, 2023

dbuezas commented Jun 18, 2023 •

edited

Loading

dbuezas commented Jul 6, 2023

AlexxIT commented Jul 6, 2023

AlexxIT commented Jul 6, 2023 •

edited

Loading

dbuezas commented Jul 6, 2023

AlexxIT commented Jul 6, 2023

dbuezas commented Jul 6, 2023 •

edited

Loading

dbuezas commented Jul 6, 2023

dbuezas commented Jul 9, 2023

AlexxIT commented Jul 10, 2023 •

edited

Loading

AlexxIT commented Jul 10, 2023

dbuezas commented Jul 10, 2023

AlexxIT commented Jul 10, 2023 •

edited

Loading

dbuezas commented Jul 10, 2023

dbuezas commented Jul 13, 2023

AlexxIT commented Jul 13, 2023

Add Digital PTZ via mouse and touch #491

Add Digital PTZ via mouse and touch #491

Conversation

dbuezas commented May 21, 2023 • edited Loading

For you to consider:

dbuezas commented May 22, 2023

dbuezas commented May 23, 2023

dbuezas commented May 24, 2023

zelo66 commented May 31, 2023 • edited by AlexxIT Loading

dbuezas commented Jun 1, 2023

dbuezas commented Jun 3, 2023

zelo66 commented Jun 4, 2023

dbuezas commented Jun 4, 2023

zelo66 commented Jun 4, 2023

dbuezas commented Jun 4, 2023 • edited Loading

zelo66 commented Jun 8, 2023

dbuezas commented Jun 9, 2023

AlexxIT commented Jun 16, 2023 • edited Loading

dbuezas commented Jun 16, 2023

1. Multiple files

2. Modern javascript syntax

AlexxIT commented Jun 17, 2023

dbuezas commented Jun 18, 2023 • edited Loading

dbuezas commented Jul 6, 2023

AlexxIT commented Jul 6, 2023

AlexxIT commented Jul 6, 2023 • edited Loading

dbuezas commented Jul 6, 2023

AlexxIT commented Jul 6, 2023

dbuezas commented Jul 6, 2023 • edited Loading

dbuezas commented Jul 6, 2023

dbuezas commented Jul 9, 2023

AlexxIT commented Jul 10, 2023 • edited Loading

AlexxIT commented Jul 10, 2023

dbuezas commented Jul 10, 2023

AlexxIT commented Jul 10, 2023 • edited Loading

dbuezas commented Jul 10, 2023

dbuezas commented Jul 13, 2023

AlexxIT commented Jul 13, 2023

dbuezas commented May 21, 2023 •

edited

Loading

zelo66 commented May 31, 2023 •

edited by AlexxIT

Loading

dbuezas commented Jun 4, 2023 •

edited

Loading

AlexxIT commented Jun 16, 2023 •

edited

Loading

dbuezas commented Jun 18, 2023 •

edited

Loading

AlexxIT commented Jul 6, 2023 •

edited

Loading

dbuezas commented Jul 6, 2023 •

edited

Loading

AlexxIT commented Jul 10, 2023 •

edited

Loading

AlexxIT commented Jul 10, 2023 •

edited

Loading