Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement accessibility support #552

Open
7 tasks
twitchyliquid64 opened this issue Oct 5, 2020 · 18 comments
Open
7 tasks

Implement accessibility support #552

twitchyliquid64 opened this issue Oct 5, 2020 · 18 comments
Labels
accessibility feature New feature or request help wanted Extra attention is needed

Comments

@twitchyliquid64
Copy link

twitchyliquid64 commented Oct 5, 2020

Background

Platforms typically mediate an interface between applications with a user interface, and accessibility tools (such as a screen reader), enabling assistive technologies:

Iced should provide the building blocks for making accessible applications, and integrate them automatically where possible.

Requirements

  • Integrate with native platform APIs for accessibility, and provide reasonable accessibility integration for widgets built-into iced.
    • Ability to communicate the widget hierarchy
    • Ability to communicate the type, state, and 'properties' of widgets in the hierarchy
    • Ability to query for a widget at a specific location.
    • Ability to provide updates of changes to widgets, or their location in the hierarchy
  • User-code should not need to use platform-specific crates.
  • User widgets should be able to provide additional information to accessibility APIs.
    • Additional or better markup/attributes
    • 'Navigations' or actions
    • Indicate relationships to logically-related widgets

Ideas

  • Automatic integration
    • Track widgets in a nested hierarchy
    • For each widget in the hierarchy, automatically determining basic information that can be communicated to accessibility APIs (ie: whether its a button/label/input etc, button/label text, focus state).
    • Communicate a representation of the hierarchy, basic widget information, and mutations to the platform-specific accessibility interface. This implementation could be shared with layout debugging and unit-testing facilities.
    • Implement hit-testing.
  • Widget-specific extensions
    • Allow widgets to provide additional markup or attributes to an accessibility API
    • Allow widgets to indicate actions/navigation options to an accessibility API (this could be implemented at a higher level as messages?)
    • Allow widgets to indicate relationships to logically-related widgets

Imho, this means we first need to implement:

  • A widget hierarchy that lives longer than layout/draw iterations
  • A stable identity for a widget in the hierarchy.

Related issues

#282

@valpackett
Copy link
Contributor

GTK/Gnome - Accessibility Toolkit (ATK)

Note: GTK4 is now using AT-SPI directly, because ATK is a barely maintained thing that only adds unnecessary indirection.

@hecrj hecrj added feature New feature or request help wanted Extra attention is needed labels Nov 27, 2020
@savente93
Copy link

Taking inspiration from semantic HTML and its accessibility: would it be an idea to make certain marker traits such as Header, Title, Section, List. These traits could then implement necessary accessibility functionalities that (hopefully) wouldn't impact crate users at all.

Personally, I think it also has the benefit of making the code more readable, just like semantic HTML.

@kaushalyap
Copy link

@hecrj Why is not this on the roadmap?

@feylinara
Copy link
Contributor

The first version of accesskit was published a few days ago, it's probably worth keeping an eye on it. Even if iced decides to go another route they're doing stuff like upstreaming stuff to winit that's necessary to elegantly implement MSAA/UIAutomation

@Alch-Emi
Copy link
Contributor

Alch-Emi commented Jan 25, 2022

I want to link in this thread, discussing accessibility in egui, and specifically Matt Campbell (@mwcampbell)'s contribution (linked) at the bottom of the thread.

The tldr is this: egui is currently trying to do integrated TTS as a stopgap for true screenreader accessibility. However, Matt has prototyped an implementation of egui that uses the accesskit beta (alpha? prototype?) to get real screenreader accessibility everywhere accesskit supports it, including a rough fork of winit to add support.

Matt also provides a brief overview of accesskit's architecture, which sounds like it will mesh really well with the elm-style architecture iced uses:

for each frame where something in the UI has changed, the application creates a TreeUpdate, which can be either a complete tree snapshot or an incremental update, and pushes it to an AccessKit platform adapter

Other highlights:

Addendum: Huge kudos to Matt Campbell and Nolan Darilek. I feel like I didn't emphasize this before editing this post, but y'all's work is wonderful and appreciated


If I may switch from "providing updates mode" to "asking questions about the projects direction mode", I might want to ask (or at least open discussion for) where iced wants/plans to take screenreader accessibility in the future. At this point, I imagine there are a few options:

  • Immediately adopt accesskit (most future proof, but means dropping linux/macos support for screenreader users until those platform adapters are released) related issue: Basic Unix adapter AccessKit/accesskit#56
  • Follow in egui's footprints by adopting a tts implementation (medium amount of work, not true accessibility, but probably the fastest path to a cross-platform accessible iced)
  • Directly interface with platform accessibility interfaces (an insane amount of work, will probably take a long time, might be lighter weight? probably shouldn't do this)
  • Something else? suggestions?

@13r0ck
Copy link
Member

13r0ck commented Jan 27, 2022

Agreed, acesskit seems promising

@mwcampbell
Copy link

@Alch-Emi Thanks for looping me in here, and sorry for the late response.

I hope to start working on AccessKit in earnest again in a few weeks. I'd call it alpha right now. If all goes according to plan, the Windows implementation should be beta-quality in a few months, and then I'll start on the Mac implementation. Other platforms will probably have to wait a while. I may be able to spend some time working on integration into iced.

@savente93
Copy link

Having accessibility would be a huge competitive advantage for me in iced as not many or even any other rust frameworks (that I'm aware of) use it at the moment. Would love to help work on this is that would be helpful :) Very happy to see this discussion!

@hecrj hecrj pinned this issue May 3, 2022
@mwcampbell
Copy link

mwcampbell commented Jul 22, 2022

A quick update: AccessKit is still Windows-only, and there are still serious limitations in the Windows implementation, most notably lack of support for text editing. But one major blocker has just been resolved: the newly published accesskit_winit crate makes it straightforward to use AccessKit with winit, without requiring changes to winit itself.

@ids1024
Copy link
Contributor

ids1024 commented Nov 9, 2022

Widget-specific extensions
Allow widgets to provide additional markup or attributes to an accessibility API
Allow widgets to indicate actions/navigation options to an accessibility API (this could be implemented at a higher level as messages?)
Allow widgets to indicate relationships to logically-related widgets

For an initial design for accessibility support, I think these are particularly important to consider (the other parts are just "implementation details", in theory).

This probably has to be done by adding methods to the Widget trait? How would the "indicating relationships" part handled with the widget model Iced uses?

I suppose Elm must have some kind of accessibility support, so that's worth looking at.

@ids1024
Copy link
Contributor

ids1024 commented Nov 9, 2022

Or actually, adding method(s) to Widget is perhaps only part of this. There are two things here:

  • How does the implementation of a widget (say, Button) define its default accessibility properties?
  • How can accessibility properties be set by the application author, to set things that aren't automatically handled by the widgets?

@ids1024
Copy link
Contributor

ids1024 commented Nov 15, 2022

It looks like https://package.elm-lang.org/packages/tesk9/accessible-html/latest is worth looking at for an elm-style way of dealing with accessibility.

@mwcampbell
Copy link

Quick update on AccessKit: The Windows adapter now has very usable text editing support, and integration into egui is now a draft PR. The Linux adapter (a pure Rust implementation of AT-SPI) is progressing nicely. Now I need to focus on the Mac adapter.

@paulGeoghegan
Copy link

Is there any updates on this?

@MichaelMcDonnell
Copy link

@paulGeoghegan, it is being worked on according to this blog post from last year.

@jackpot51
Copy link

@hecrj we have carried a fork of iced with accesskit integration for a while, for use in COSMIC. I believe the changes are in a good place to be upstreamed, and if that is desired, then my team (cc @mmstick @wash2) can work to clean them up and submit a PR.

@hecrj
Copy link
Member

hecrj commented Jun 20, 2024

@jackpot51 Sounds good to me! Accessibility is planned for the release after the next one, which should happen relatively soon.

My idea was to bring #1849 up to speed, but I imagine that is quite outdated so I definitely will appreciate any help in that regard.

@wash2
Copy link
Contributor

wash2 commented Jun 20, 2024

Ya, I will work on updating that PR 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accessibility feature New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests