New keycode macro (XP) for shifted character pairs using UNICODEMAP + bug fixes and improvements #4803

vomindoraan · 2019-01-08T19:47:57Z

Add XP(i, j), a keycode macro¹ like UC and X that allows users to easily input pairs of Unicode characters. The most common use case for this is inputting lower and upper case versions of a character, for example å/Å, based on whether Shift or Caps Lock are active.

Also improve parts of the existing Unicode code, for example by adding a unicode_input_cancel() function and fixing an off-by-one error in a range check in process_unicode(). The latest version of WinCompose increased the range of supported code points, so I implemented this as well.

I've also made some substantial improvements to the Unicode docs in this PR.

Description

Currently, to add a key that sends å or Å you would need to define a custom keycode, then handle the Shift/Caps logic yourself and output the corresponding character. XP makes this a lot easier: you just need to add XP(i, j) to your keymap, where i is the index of the first (lower case) and j the index of the second (upper case) character in unicode_map.

This was done by expanding the QK_UNICODEMAP range from 0x8000–0x83FF to 0x8000–0xFFFF. This makes the UNICODEMAP range equal to the range for UNICODE. This shouldn't be an issue as the two are feature flagged and guaranteed to be mutually exclusive. It also doesn't affect existing keycode ordering and keymap storage in EEPROM as there can be no keycodes above 0xFFFF anyway.

The two character indices are then packed into the 16-bit keycode as follows:

11ii iiii ijjj jjjj
├┘└───┬───┘└──┬───┘
│     │       7 bits for second char index (0–127)
│     7 bits for first char index (0–127)
Denotes QK_UNICODEMAP_PAIR range

process_unicodemap() unpacks this value, checks the Shift and Caps states, then outputs the appropriate character from unicode_map.

Changes

Add new keycode macro¹ XP(i, j) for the feature.
Expand UNICODEMAP Quantum keycode range² from 0x8000–0x83FF to 0x8000–0xFFFF.
Fix bugged range check (OB1 error) in process_unicode() (> was being used instead of >=); make this check more robust by comparing keycodes with the upper bound as well.
Replace unicodemap_input_error() (unused) with more generic unicode_input_cancel().
Tweak how unicode_input_start() and unicode_input_finish() are called in process_unicodemap() as part of the XP implementation.
Tweak Unicode headers (add missing declarations, newlines; remove redundant includes).
Rename UNICODE_OSX_KEY → UNICODE_KEY_OSX and UNICODE_WINC_KEY → UNICODE_KEY_WINC for consistency and ease of search.³
Add UNICODE_KEY_LNX since the input combo for Linux may have changed in newer versions of IBus.³
Update related documentation.

_{¹ Doesn't affect keycode ordering and EEPROM keymap storage since it's a macro.}
_{² Doesn't affect keycode ordering and EEPROM keymap storage since the space above 0x8000 is already reserved for UNICODE/UNICODEMAP.}
_{³ Slightly outside the scope of this PR, but I figured I might as well do it now.}

Thanks @asmand for giving me the idea for this feature and providing the use case.

Tested and confirmed working on Windows, Mac and Linux. Currently it doesn't work correctly on Linux if Caps Lock is on, but it turns out that's a preexisting issue with how we send Unicode input. I will open an issue and make a separate PR for that.

Types of changes

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document. (https://docs.qmk.fm/#/contributing)
I have added tests to cover my changes.
I have tested the changes and verified that they work and don't break anything (as well as I can manage).

…s calculated

…el()

vomindoraan · 2019-01-08T23:14:22Z

I will retest these latest changes on Mac and Linux tomorrow.

vomindoraan · 2019-01-08T23:43:18Z

I renamed the keycode range from QK_UNICODEMAP_SHIFT to QK_UNICODEMAP_PAIR, but I'm not sure if XP(i, j) is better than XS(i, j).

Edit: I decided to go with XP.

vomindoraan · 2019-03-24T04:49:53Z

Updated docs per review feedback.

@jackhumbert @drashna Any chance you could take a look at this PR again and merge it?

vomindoraan · 2019-03-24T14:52:30Z

This PR should probably be labeled documentation as well, given the large set of changes/improvements to the Unicode docs.

skullydazed

I'd still like @jackhumbert to weigh in on the expanded keycode issue.

vomindoraan · 2019-04-21T10:50:05Z

@jackhumbert Could you please take a look at the keycode ranges so this can be merged? It's the only thing that's left to do.

vomindoraan · 2019-05-03T10:48:15Z

Retested with latest master merged in, and confirmed working on Windows, Mac and Linux (thanks to my wonderful coworkers who let me use their machines).

… bug fixes and improvements (qmk#4803) * Expose unicode_saved_mods * Add UNICODEMAP shift pair functionality and XS keycode * Add XS to keycode reference documentation * Pick pair index based on both Shift and Caps Lock state * Add XS to Unicode feature docs * Clean up process_unicode* headers * Extract unicode_map index calculation into function * Pick pair index as XOR rather than OR of Shift and Caps states * unicode_input_start() has to be called before the unicode_map index is calculated * Replace unicodemap_input_error() with more generic unicode_input_cancel() * Replace register+tap+unregister with tap_code16(LCTL(LSFT(KC_U))) * UNICODE_OSX_KEY → UNICODE_KEY_OSX, UNICODE_WINC_KEY → UNICODE_KEY_WINC * Make keycode range checks more robust * Fix keycode range checks for different input modes * Add UNICODE_KEY_LNX, update docs * QK_UNICODEMAP_SHIFT → QK_UNICODEMAP_PAIR * XS → XP, update docs * Tweak Unicode docs * Use recently added MOD_MASK_SHIFT and IS_HOST_LED_ON helpers * Update Unicode table in docs/keycodes.md * Update Unicode docs per review comments * Replace references to Mac OS X with macOS in Unicode docs * As of v0.9.0, WinCompose supports all possible code points * Expand descriptions in XP docs * Update keycode table and cycling docs * Further expand cycling docs

vomindoraan added 11 commits January 8, 2019 14:01

Expose unicode_saved_mods

62530c8

Add UNICODEMAP shift pair functionality and XS keycode

e2974cf

Pick pair index based on both Shift and Caps Lock state

cb50cc6

Add XS to keycode reference documentation

13bb673

Add XS to Unicode feature docs

1b0fe41

Clean up process_unicode* headers

53db742

Extract unicode_map index calculation into function

96a4558

Pick pair index as XOR rather than OR of Shift and Caps states

2e149ad

unicode_input_start() has to be called before the unicode_map index i…

39c7e1a

…s calculated

Replace unicodemap_input_error() with more generic unicode_input_canc…

5393c34

…el()

Replace register+tap+unregister with tap_code16(LCTL(LSFT(KC_U)))

2bc2f3f

vomindoraan changed the title ~~New keycode macro (XS) for shifted character pairs in UNICODEMAP~~ New keycode macro (XS) for shifted character pairs using UNICODEMAP Jan 8, 2019

vomindoraan added 3 commits January 8, 2019 21:09

UNICODE_OSX_KEY → UNICODE_KEY_OSX, UNICODE_WINC_KEY → UNICODE_KEY_WINC

9e725d1

Make keycode range checks more robust

9858c84

Fix keycode range checks for different input modes

e1633f2

vomindoraan force-pushed the unicodemap_shift branch from 8117441 to e1633f2 Compare January 8, 2019 22:42

Add UNICODE_KEY_LNX, update docs

ebabade

vomindoraan changed the title ~~New keycode macro (XS) for shifted character pairs using UNICODEMAP~~ New keycode macro (XS) for shifted character pairs using UNICODEMAP + bug fixes and improvements Jan 8, 2019

QK_UNICODEMAP_SHIFT → QK_UNICODEMAP_PAIR

3f6fcb7

XS → XP, update docs

5cf507c

vomindoraan changed the title ~~New keycode macro (XS) for shifted character pairs using UNICODEMAP + bug fixes and improvements~~ New keycode macro (XP) for shifted character pairs using UNICODEMAP + bug fixes and improvements Jan 9, 2019

Tweak Unicode docs

4630890

vomindoraan force-pushed the unicodemap_shift branch from 73a69a3 to 4630890 Compare January 10, 2019 14:53

drashna requested review from drashna, skullydazed, jackhumbert, ezuk and fredizzimo January 10, 2019 19:42

vomindoraan force-pushed the unicodemap_shift branch from b2c9e96 to aab9d25 Compare March 24, 2019 04:48

vomindoraan added 2 commits March 24, 2019 13:52

Update Unicode docs per review comments

a10ee21

Replace references to Mac OS X with macOS in Unicode docs

fce7c23

vomindoraan force-pushed the unicodemap_shift branch from d6cd65a to fce7c23 Compare March 24, 2019 12:54

vomindoraan added 2 commits March 24, 2019 14:32

As of v0.9.0, WinCompose supports all possible code points

71f640d

Expand descriptions in XP docs

6b98877

vomindoraan force-pushed the unicodemap_shift branch from 1b9d39d to 6b98877 Compare March 24, 2019 14:10

vomindoraan added 2 commits March 24, 2019 15:22

Update keycode table and cycling docs

e1654b8

Further expand cycling docs

85fa4b8

skullydazed approved these changes Apr 5, 2019

View reviewed changes

Merge branch 'master' into unicodemap_shift

cc614ed

jackhumbert approved these changes May 3, 2019

View reviewed changes

drashna merged commit bdc8d89 into qmk:master May 3, 2019

vomindoraan deleted the unicodemap_shift branch May 3, 2019 16:39

drashna mentioned this pull request May 6, 2019

New keycode macro (XP) for shifted character pairs using UNICODEMAP +… zsa/qmk_firmware#45

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New keycode macro (XP) for shifted character pairs using UNICODEMAP + bug fixes and improvements #4803

New keycode macro (XP) for shifted character pairs using UNICODEMAP + bug fixes and improvements #4803

vomindoraan commented Jan 8, 2019 •

edited

Loading

vomindoraan commented Jan 8, 2019

vomindoraan commented Jan 8, 2019 •

edited

Loading

vomindoraan commented Mar 24, 2019

vomindoraan commented Mar 24, 2019 •

edited

Loading

skullydazed left a comment

vomindoraan commented Apr 21, 2019

vomindoraan commented May 3, 2019 •

edited

Loading

New keycode macro (XP) for shifted character pairs using UNICODEMAP + bug fixes and improvements #4803

New keycode macro (XP) for shifted character pairs using UNICODEMAP + bug fixes and improvements #4803

Conversation

vomindoraan commented Jan 8, 2019 • edited Loading

Description

Changes

Types of changes

Checklist

vomindoraan commented Jan 8, 2019

vomindoraan commented Jan 8, 2019 • edited Loading

vomindoraan commented Mar 24, 2019

vomindoraan commented Mar 24, 2019 • edited Loading

skullydazed left a comment

Choose a reason for hiding this comment

vomindoraan commented Apr 21, 2019

vomindoraan commented May 3, 2019 • edited Loading

vomindoraan commented Jan 8, 2019 •

edited

Loading

vomindoraan commented Jan 8, 2019 •

edited

Loading

vomindoraan commented Mar 24, 2019 •

edited

Loading

vomindoraan commented May 3, 2019 •

edited

Loading