User interface

[koenvaneijk]

Following issue #18 raised by @martinszeltins, please use this discussion to talk about what the user interface should look like.

[martinszeltins]

Thanks @koenvaneijk

I would like to raise the previous question and get your thoughts on this. I might be able to contribute with the frontend and UI/UX part.

Can this be implemented as a frontend that is seperate from the backend? For example using the Nuxt framework that communicates with the openrecall backend (using HTTP protocol)? We would also need to agree on the API used for the communication. I assume this would require changes on the backend, right?

[VasigaranAndAngel]

Sorry to interrupt. I'm a bit confused about why we need Nuxt and web stuff to make a UI for a local application.

[martinszeltins]

It's not an OS native app but a web app. At least in it's current implementation.

[koenvaneijk]

I'm personally not a big fan of introducing another ecosystem like npm with a separate build pipeline for the frontend. One reason being is that we want to keep the project easily maintainable in the future, I understand the temptation of a shiny JavaScript framework and I agree that it can be a lot faster at delivering value than say Python > Flask > HTML with Bootstrap, however, the latter have been around forever and don't require a gazillion npm dependencies or build/compile step.

I think for the OpenRecall project the following pillars are critical:

• Auditability: everyone with basic Python (or any other high level language) knowledge should be able to review the code to verify (and thus trust it).
• Maintainability: we want to keep the solution maintainable, and mitigate the risk of abandonware in dependencies.
• Security: a smaller dependency footprint translates into better security with regards to supply chain attacks (hijacked dependencies such as what we saw with xz can happen to any small/seemingly insignificant package).

TL;DR: Yes, the UI should be improved and look more appealing, as this will undeniably drive further adoption of the project. I think we should look at how far we can get with just HTML, CSS and some minor JavaScript (again, auditability) for interactivity.

Open for discussion!

[VasigaranAndAngel]

Another question arises - will the application always be written in Python or will it eventually switch to Rust or C/C++ for performance reasons?

This application won't be overly complex, so I don't think there will be a need to write code in multiple languages for performance reasons. PySide serves as a bridge to the Qt framework, which is implemented in C++. As a result, there is minimal impact on performance.

The more I think about this, the more I lean toward having a reference implementation in plain old HTML/JS, with the community building different front ends on the same OpenRecall API. Thoughts?

That's a good idea. ❤️

[martinszeltins]

This application won't be overly complex

Maybe. Maybe not. Every new app starts out simple thinking it won't be complex. But I think it is good to plan for the future and be able to scale up if needed.

[m5x5]

The more I think about this, the more I lean toward having a reference implementation in plain old HTML/JS, with the community building different front ends on the same OpenRecall API. Thoughts?

1. I understand where you are coming from, but this seems like moving the responsibility to others, that might not have the security in mind like you have. Of course you might not be the one who's going to check the FE code for security concerns, but at least we have the code in the same repository and have the ability to administer what plugins are installed.

2. If you decide to only create a strongly typed api, with just a basic frontend, and try to delegate the frontend building to the community new people will have to search for the frontend they want to use and if they don't, it's possible that they are missing out on features that someone else built.

My advice: Make a decision for a frontend, don't delegate it, it'll be less confusion, quicker to setup and you'll have a better control over the security.

Auditability: everyone with basic Python (or any other high level language) knowledge should be able to review the code to verify (and thus trust it).
You'll have to decide between features, readability and security there.

• Increase in Features = Less Readability = (Less Security) (increase in codelength)
• Increase in Abstraction to improve readability does not necessarily mean less security. (a good abstraction like TailwindCSS (for styling) has been around for a while.

My suggestion: If FE becomes more complicated make sure to use an abstraction like React or some other FE Framework. It's safer to use already established bullet proof code than trying to build a framework yourself. As soon as you notice that an abstraction is needed choose a robust and scalable one, otherwise Auditability will suffer.

[martinszeltins]

@m5x5 The idea is to ship with a basic implementation of a frontend and allow the community to create "custom" frontends. This is what a lot of open source programs do and it works quite well, for example:

1. Git has multiple GUIs written for it (GitHub is one of them)
2. ffmpeg also has multiple GUI apps available written on top of ffpmeg.
3. MySQL - phpMyAdmin, adminer, SequelPro, HeidiSQL, DBeaver etc.
4. Redis has multiple GUI clients
5. Docker has lots of GUI apps too.

All of those frontends communicate with the backend via some kind of protocol or API and do not pose a real security threat (especially web based ones). All security measures are implemented on the backend.

And perhaps if someone creates a very popular frontend that everyone wants to use then it could be shipped as the default one eventually (or recommended at installation).

[m5x5]

@martinszeltins Yeah, you're right I didn't think of that, that's a great point.
I guess now we'll have to discuss what goes into a "basic" frontend.

[martinszeltins]

@koenvaneijk I agree that auditability and maintainability are important. As a frontend developer (former backend dev), I can resonate with wanting to keep things simple and maintainable. But I believe that modern frontend frameworks like Nuxt can actually help improve auditability, maintainability, and security in the long run. It will also help with attracting more contributors to the project from the frontend / designer community. Believe me, nobody wants to work with raw HTML, bootstrap in 2024 (I mean frontend devs). To frontend devs it sounds like wanting to write it in assembly instead of Python (just to keep it simple?) :)

Just a few thoughts from my side on these issues:

1. Attracting frontend talent
   By using modern frontend frameworks, we can attract more frontend developers to the project. Like I said, no frontend dev wants to work with raw HTML and Bootstrap in 2024. Currently, the frontend part is tightly coupled with Python and no frontend dev will want to touch that. There are many talented frontend developers out there who can contribute AND maintain the frontend part of the project. The frontend community is one of the largest. It also hugely contributes to developer happiness, it is a joy to work with modern frameworks. Why alienate the frontend community?

2. Auditability
   I agree that not everyone will be able to understand the frontend code. But not everyone will be able to understand Python either. Does that mean we should write in assembly? We need to think long-term about the project. As the project grows and gets larger, not everyone will be able to understand every language, every tool, every framework used in the project. If the project attracts frontend developers, they will be able to audit and maintain it. And we can (and should) always write good documentation to help with that.

3. Best Practices
   Working with raw HTML, JavaScript, Bootstrap, jQuery is no fun (ask any frontend dev). It's error prone, hard to maintain, and doesn't follow best practices. Without TypeScript it is very difficult to read the code because you don't know what type of data you are working with. Without Tailwind it is very difficult to work with and maintain the CSS in the long run. Frontend frameworks introduce structure and best practices that make the codebase easier to understand and maintain. How is one ball of html/css/js mess more maintainable than a well structured, strongly typed, modular codebase with good architecture following best programming practices (SOLID, DRY etc.?)

4. Maintainability
   If we go with a proper framework, the project will actually be more maintainable. I have fallen into this trap multiple times in the past myself. I have started projects with just raw html/js thinking it will be enough and quickly realized that my code was getting out of hand and impossible to maintain. I wasn't able to understand my own code anymore. That is the whole reason why frameworks exist - they make the project maintainable in the long run (and they are a joy to work with). If you need some common components - you can easily grab a component lib that has everything you need. No need to reinvent the wheel.
   Also, consider the speed of development. Modern frontend frameworks have HMR (hot module replacement). As soon as you make one change in your source code, the result is immediately reflected in the browser. You see your changes on the screen immediately instead of having to rebuild / refresh / restart anything. If you need some functionality - just install a module and you're done. To implement the same things from scratch takes much more time and effort. A frontend dev with proper tooling can move 10x faster. Why wouldn't we want that?

5. Security
   The npm ecosystem has matured a lot since the early days and has tools to audit packages for security vulnerabilities so I don't think this is much of a concern. Also all frontend code runs in a sandboxed environment so there is no real risk for the end user.

I can imagine in the early days when high-level languages like C, C++, Python started to appear people probably wanted to keep writing in assembly to keep the code simple and easy to understand. But nowadays nobody writes in assembly anymore because it is not maintainable. People have realized that high-level languages and frameworks are the way to go. Nobody creates raw PHP apps today, instead everyone uses frameworks like Laravel, Symfony etc. I think the same applies to frontend development. We should use modern frontend frameworks to make the project maintainable and attractive to frontend developers.

Do you think I have some valid points? I am also open for discussion and would like to hear your thoughts on this.

[m5x5]

There are tradeoffs for every package installed, sure we have them on the python side and we should audit them. But does it mean we should do the same on the frontend? I still see it as optional to deliver JavaScript from other NPM packages. We could use good old Handlebars with TailwindCSS to create the frontend and for the interactivity we'll write some JavaScript.

Don't you think this is an environment that one can create a decent but still safe website in?
What security enhancements do blobs offer? Aren't malicious actors still able to grab the rendered image tag?

On the topic of auditing the network tab: Is it really possible to audit this reliably. What if the data exchange doesn't happen any time soon but rather every month or so?

I'm sorry for pushing so hard against modern FE frameworks and tools, I also love to use them, but I'm just trying to make sure we still try to do our best to keep this as secure as we can without compromising on a bit of maintainability security on this project.

[martinszeltins]

Well, like I said if we wanted to be 100% safe then we'd need to audit every line of code in Python and Python packages that are used and their dependencies etc. We would have to audit the browser code and every npm package.

The frontend is the least insecure part of the project. It can be a lot worse if there is malicous code on the backend side since that could do a lot more damage. So the best way would be to not use Python or python packages but write it without using any 3rd party code. We could use pure C but cannot rely on standard library so those parts would need to be written in assembly.

The frontend is more secure because it has limited access to resources. We can even obfuscate the snapshots by drawing them onto a canvas or using a blob or any other method. Also, we could prevent 3rd party code from making requests by overwriting XHR and fetch objects.

I feel like frontend is the least of our problems. Instead we should be much more concerned about the backend that has access to EVERYTHING and can do basically everything with the data. Wouldn't it make sense to write the backend in pure C/Assembly?

[m5x5]

Right now I was just focusing on reducing the attack surface on the frontend, but sure we could also overwrite the objects that are responsible for requests, it would make it a little more complicated.

Are you serious about rewriting the backend in C? I'm not sure anymore.

[martinszeltins]

Are you serious about rewriting the backend in C? I'm not sure anymore.

What do you think about that? That would be a lot more secure.

[m5x5]

I'm a web developer, so I don't have experience in lower level programming. But I can imagine it being difficult to rewrite what we've got in C. Especially interactions with the ML Model and making it cross platform will be challenging.

[m5x5]

So what are going to be the next steps @koenvaneijk? You see our discussion, what's your decision? Are we going to have a basic CSS / HTML Frontend with an API so that the community can create more frontends or what do you suggest?