Native Speech History in NVDA

[CyrilleB79]

Is your feature request related to a problem? Please describe.

Here are some user stories

User story 1

As an NVDA user,
I want to quickly repeat the last spoken message,
so I can ensure I heard it correctly. For example, when the last message was obscured by a sudden noise, such as a door slamming.

User story 2

As an NVDA user,
I want to navigate through a list of recently spoken messages,
so I can revisit important information announced earlier such as notifications.
For example, I can continue to do what I am doing for a few seconds and then hear the last received e-mail notification. Depending on its content I can then switch to Outlook immediately (urgent answer needed) or later.

User story 3

As a visually impaired user using both Magnifier and NVDA,
I want to be able to revisit last spoken message without being bothered by the speech viewer window always on top of my screen.

User story 4

As an NVDA or add-on developer or tester,
I want to be able to copy efficiently GUI labels or last spoken messages and paste them somewhere else, e.g. GitHub issue or PR description, answer on mailnig lists, writing documentation, etc. To be efficient, I do not wish to have to ensure the speech viewer is active and to alt+tab to it.

User story 5

As an NVDA user,
I want to be able to copy easily what I am told (NVDA messages, elements from applications) to ask for support.

User story 6

As an NVDA user,
I want to be able to revisit and copy last spoken messages when I am working with a sighted person, without having the speech viewer covering a part of the screen.

User story 7

As an NVDA user,
I want to be able to do a screenshot to ask support without having to think at hiding the speech viewer before.

Why not use the speech viewer?

• Opening the speech viewer and copying from it requires more actions (NVDA menu, alt+tab, navigatino in the viewer, selection, copy)
• Having the speech viewer on top of other windows is useful when making a demonstration so that sighted people realize what NVDA say. But when a sighted colleague just want to work with me, they do not need to know all what NVDA says and having the speech viewer on top is inefficient.

Describe the solution you'd like

The UX provided by Speech history review and copying by @jscholes is very satisfying and efficient. Only the native gestures of this add-on (shift+f11, shift+f12 and f12 ) should be modified because, not using NVDA key, they conflict with many native application shortcut. We may use the gestures provided by the Speech History Explorer add-on, a fork of the previous one, provided by David ACM.

So the feature request is as follows:

• NVDA keeps in memory a buffer of the last spoken phrases, e.g. 500 phrases (or configurable)
• NVDA+control+f12: copy the last spoken phrase in the clipboard
• NVDA+shift+f11/f12: circulate in the last spoken phrase buffer to hear the previous / next phrase. After having heard a phrase, pressing NVDA+control+f12 can copy it to clipboard.

E.g. if you hear two notifications in a row and want to copy the penultimate to clipboard: press NVDA+shift+f11 and then NVDA+control+f12.

In a second time, improvements can be imagined such as:

• a GUI to review the buffer of last speech (as in Speech History Explorer)
• a "repeat more slowly" command such as when listening to Google Translation (on the web)
• if needed: configurability of parameters as in the add-on such as: trim spaces before or after copied text, number of phrases in the buffer, etc.

Describe alternatives you've considered

Users can keep on using add-ons:

• Speech history review and copying by @jscholes, available in the Add-on Store
• Speech History Explorer by @davidacm: a fork of the previous one, unfortunately not present in the Add-on Store, with additional features (e.g. a GUI to review the speech and copy multiple messages at once)

Users can also use the native speech viewer, with more actions needed, but the goal of this feature is to have something more efficient.

Additional context

• Both speech history add-ons seem to be widely used in the community and have been requested various times to be integrated in core. The one from James Scholes is in the community for at least 10 years, I guess
• Around 2021, these add-ons have had issues to make them compatible with NVDA (say all not stopping), causing the add-on not too be available or compatible with newer NVDA version (or beta). A native integration would prevent such inconvenience.
• Other screen readers have such feature, sometimes partly:
  • Jaws' Speech history
  • Narrator: command to repeat last phrase (Narrator+V)
  • Talkback: a command to copy last spoken phrase
  • I do not know about Linux and Apple world
• Other issues had already been written about it in the past (speech history #6905, speech history being built into the screen reader by default #16644, The speech history add-on being built in to nvda #17325, Integration of speech history functionality to NVDA core #18349), but have been closed because the need was not correctly described and / or the template not used. Hope this one does the job; I've taken time to try to make a correct description.

[CyrilleB79]

Cc @gogabs

[jscholes]

Just to note: the latest version of Speech History has a browse mode window for reviewing the history entries, searching using NVDA's standard find feature, and copying multiple entries to the clipboard. This aligns it more with the Explorer fork.

[amirmahdifard]

I agree 100% Hope someone can be in faver of implementing this.

[CyrilleB79]

This issue has been triaged.

@jscholes or @davidacm, would you be interested in integrating the speech history feature in NVDA?

[jscholes]

@CyrilleB79 On the assigned p5 priority, the description of which states:

Other feature requests affecting a small subset of users

I think there's a bit of a conceptual contradiction here. Either:

• Speech history is useful to a large swath of NVDA's user base, in which case it might make sense to have it in core and it should probably not be a p5. Or...
• It's only of interest to a small subset of people, in which case the p5 label is accurate, but it's probably fine as an add-on and would be a waste of development time for someone and review time for NV Access.

The community response to the Speech History add-on, download counts, and number of issues that have been created requesting its integration suggest that it would fit more appropriately under p4:

Useful popular feature request or enhancement

However, I'm not sure what data NV Access or @SaschaCowley specifically has to suggest that p5 is a better fit.

Regardless, I currently don't have the bandwidth to work on an integration into core. It's also not particularly clear to me which feature(s) of Speech History would and would not be welcome. Before even thinking about working on a port of the existing add-on, I'd like to know how much of that work is likely to be approved, and any aspects of the add-on's user experience (beyond the much mentioned default shortcut keys) that NV Access and/or the community feel are not appropriate.

[davidacm]

Hi @CyrilleB79, I will work on this this weekend or before. Discussing ideal keyboard commands would be to open a topic of never ending. But until there is a PR, the discussion won't be happen properly.

[davidacm]

btw. I will use the approach of the original add-on to show the list of messages, it's easier to implement than my implementation that was developed when showing html content was not available. If someone disagree with this, please comment here.

[SaschaCowley]

Hi @davidacm,
Please note that we think a "repeat last utterance" command would cover most use-cases here fine, while avoiding most of the privacy and performance problems with a more extensive approach. I suggest only opening a PR with such a command to begin with.

[davidacm]

Hi @SaschaCowley. For me, is not enough with the last utterance. And other users could have the same situation. I could set the numer of saved utterances to 1 as default (so the user could increase it if needed) but if a list of history utterances won't be allowed, then I prefer not to implement this on the core of NVDA.
An option to disable this feature could be added.
But I don't see the problem having a list of utterances spoken, other SRs have it as I can remember; like JAWS.

[CyrilleB79]

@SaschaCowley:

1. Can you answer @jscholes's very relevent comment (Native Speech History in NVDA #18400 (comment))? I fully agree that it should be a p4. The request is very popular and does not affect a minimal subset of users, as evidenced by the multiple requests (speech history #6905, speech history being built into the screen reader by default #16644, The speech history add-on being built in to nvda #17325, Integration of speech history functionality to NVDA core #18349 and I do not mention messages on the mailing lists).

2. Regarding the "Repeat last utterance" solution, it is very minimalist and incomplete. It only covers User story 1. It does not cover User stories 2, 4, 5 and 6, either because you cannot copy last spoken text, or because you only use a 1-utterance buffer (last one), or both..

3. Can you be more specific about security concerns? * NVDA's log at IO or DEBUG already contains an history of the speech. If you are concerned by speech memorized by default, we may disable the 500-utterence buffering by default and allow the user to enable it, mentioning in the User Guide the risk of enabling it, as we do for log level. We may also add a Warning dialog when enabling it if needed.

   • Outside of NVDA, there are other experiences memorizing potentially sensitive information. Do you consider them security issues?
     • Speech history in Jaws
     • Windows clipboard history

[SaschaCowley]

@jscholes

The community response to the Speech History add-on, download counts, and number of issues that have been created requesting its integration suggest that it would fit more appropriately under p4:

Where are you getting analytics on its uptake? We currently do not have such a mechanism in NVDA.

However, I'm not sure what data NV Access or @SaschaCowley specifically has to suggest that p5 is a better fit.

Our internal discussion of this issue lead us to the conclusion that a repeat/copy last utterance command would meet most use-cases, while being simpler, more privacy-preserving, and less resource-intensive.

Regardless, I currently don't have the bandwidth to work on an integration into core. It's also not particularly clear to me which feature(s) of Speech History would and would not be welcome. Before even thinking about working on a port of the existing add-on, I'd like to know how much of that work is likely to be approved, and any aspects of the add-on's user experience (beyond the much mentioned default shortcut keys) that NV Access and/or the community feel are not appropriate.

As we (myself and, to the best of my knowledge, Mick) don't use the add-on, that's not something we can comment on. When proposing an integration, it is up to those making the proposal to lay out the features and UX they are proposing.

---

@CyrilleB79

2. Regarding the "Repeat last utterance" solution, it is very minimalist and incomplete. It only covers User story 1. It does not cover User stories 2, 4, 5 and 6, either because you cannot copy last spoken text, or because you only use a 1-utterance buffer (last one), or both..

Apologies for not being clear enough. The proposal is to have a "repeat last utterance" gesture that would work similarly to many of our other gestures, in that pressing it multiple times would allow you to copy the utterance to the clipboard. I believe this meets most of the listed user stories.

3. Can you be more specific about security concerns?

Certainly. Foremost I think this is a privacy concern. Consider: I log in to a website or app with sensitive information (banking, government services, medical services etc). After I am done, I log out of the website or app. However, the information that I accessed is now remembered by NVDA. This could potentially become a security issue if secrets are stored. For instance, if the user reads a password from their password manager.

• NVDA's log at IO or DEBUG already contains an history of the speech.

And the default log level is info, and we encourage users only to increase the log level when asked, and only for as long as necessary.

• If you are concerned by speech memorized by default, we may disable the 500-utterence buffering by default and allow the user to enable it, mentioning in the User Guide the risk of enabling it, as we do for log level. We may also add a Warning dialog when enabling it if needed.

Yes, I think if this is added to core, we would require it to be disabled by default and to have a customisable buffer length.

   * Outside of NVDA, there are other experiences memorizing potentially sensitive information. Do you consider them security issues?
     
     * [Speech history](https://support.freedomscientific.com/teachers/lessons/5.5.1_SpeechHistory.htm) in Jaws
     * Windows clipboard history

Possibly security, certainly privacy unless they're optional, and prefferably opt-in. Windows' clipboard history is opt in, to the best of my knowledge.

[davidacm]

Hi there, I won't be working on this feature as I said before. The idea behind moving a feature from an add-on into the core is to make it accessible to all users and to eventually discontinue maintenance of the add-on itself.
Every time I've worked on implementing a new feature, the final result ends up being completely different, and users still end up needing the add-on anyway.
NVDA isn't a mobile screen reader, where simplicity is the top priority. It's a screen reader that should aim to make things easy and efficient for all users, including expert users who need maximum efficiency and customization.
I could implement everything that's been discussed here, including a warning dialog when speech history is activated, but I'm afraid most of the feature would be rejected. It's just not worth the effort.
Frankly, I find Windows' clipboard history to be far more insecure. People tend to store sensitive information there. I mean, I was able to enable it in seconds on my work laptop, which is packed with a lot of security measures.
Software needs to be open in order to evolve. Sometimes I wonder if NVDA's rigid structure and lack of flexibility have caused it to stagnate compared to other tools. Most of the new features aren't things people use daily, so I honestly don't understand the current prioritization system.