TextView

[fearless]

Also found out about this Private Character Editor (eudcedit.exe) which comes with every version of windows: https://www.makeuseof.com/tag/create-your-own-fonts-symbols-characters-windows/

[Biterider]

Hi
I have never heard of this "Private Character Editor" before.
Incredible that it has always been there. 

Biterider

[Biterider]

Hi
Searching for more information I found a useful and free tool to visualize the Unicode mapping.
https://www.babelstone.co.uk/Software/BabelMap.html

Biterider

[Biterider]

Hi
I spent some time looking for a solution to the above issue.
Since nothing really worked and there was no useful information on the internet, I tried to find out for myself.

I use the DrawText API and the "Wingding 2" font to display some symbol characters in the TextView control. In particular, I was irritated to see missing glyphs near character 128 and above (empty square). My hope is that if I can solve this problem, I will have the solution for Liaomi's system too.

I started experimenting with the ExtTextOut API. It has a very useful flag called ETO_GLYPH_INDEX that I used to output all the characters in the font in sequence. To my surprise, all of the glyphs we can see in the "font insertion dialogbox" from a MS application were there, even when DrawText refuses to display them.
I suspected that DrawText was doing some sort of mapping when using some particular fonts. 
I wrote a short program to show all characters wider than a predefined threshold and in the range 0000 to FFFFh and voilá. All the characters were there, but in very different positions!
I noticed that in the range from F000h to F0FFh all characters are in the correct order. That said, when we "and F000h" to the character value, we get the correct glyph from DrawText.

I don't have an explanation why this mapping is happening and I don't see any correlation with the Unicode BMP. 

This analysis is only valid for some sort of fonts. Other symbol fonts like Symbol, Webdings, Wingding, Wingding 3, behave the same way, while other regular fonts do not.

Changing the the charset to SYMBOL_CHARSET doesn't have any effect on the mapping.

If anyone has a clue on this topic, please post it here.

I changed the demo application to take this behavior into account. Additionally I show the characters in the range 0020h-00FFh and F020-F0FFh.
I was intrigued to see if the problem was gone on Liaomi's PC.

Biterider

[jj2007]

Win7-64

[LiaoMi]

Hi
I spent some time looking for a solution to the above issue.
Since nothing really worked and there was no useful information on the internet, I tried to find out for myself.

I use the DrawText API and the "Wingding 2" font to display some symbol characters in the TextView control. In particular, I was irritated to see missing glyphs near character 128 and above (empty square). My hope is that if I can solve this problem, I will have the solution for Liaomi's system too.

I started experimenting with the ExtTextOut API. It has a very useful flag called ETO_GLYPH_INDEX that I used to output all the characters in the font in sequence. To my surprise, all of the glyphs we can see in the "font insertion dialogbox" from a MS application were there, even when DrawText refuses to display them.
I suspected that DrawText was doing some sort of mapping when using some particular fonts. 
I wrote a short program to show all characters wider than a predefined threshold and in the range 0000 to FFFFh and voilá. All the characters were there, but in very different positions!
I noticed that in the range from F000h to F0FFh all characters are in the correct order. That said, when we "and F000h" to the character value, we get the correct glyph from DrawText.

I don't have an explanation why this mapping is happening and I don't see any correlation with the Unicode BMP. 

This analysis is only valid for some sort of fonts. Other symbol fonts like Symbol, Webdings, Wingding, Wingding 3, behave the same way, while other regular fonts do not.

Changing the the charset to SYMBOL_CHARSET doesn't have any effect on the mapping.

If anyone has a clue on this topic, please post it here.

I changed the demo application to take this behavior into account. Additionally I show the characters in the range 0020h-00FFh and F020-F0FFh.
I was intrigued to see if the problem was gone on Liaomi's PC.

Biterider

Hi Biterider,

very cool, great job! You can write an article, because even in windows there is a similar glitch with the display of squares. The picture shows that everything works fine. But I could not increase the height of the window 



@jj2007 - did you manually change the height of the window? 
P.S> Ah, now I see you glued two pictures together  Your test also has problems with displaying the font. But in the modification, all these problems have now been eliminated.

[FORTRANS]

Hi,

   Windows 10.  Cropped screenshot.

Steve

[jj2007]

@jj2007 - did you manually change the height of the window? 
P.S> Ah, now I see you glued two pictures together  Your test also has problems with displaying the font. But in the modification, all these problems have now been eliminated.

PaintShop 4.1 is my friend

[nidud]

deleted

[Biterider]

Hi
Thank you for testing. It seems to work fine, but it remains a mystery why Liaomi's system works differently. 

@nidud: thanks, I'll check it.

Biterider

[LiaoMi]

but it remains a mystery why Liaomi's system works differently. 

Hi Biterider,

Prologue>

Version 12.0.0 has been superseded by the latest version of the Unicode Standard - https://unicode.org/versions/Unicode12.0.0/
All Chapters and Appendices Together:
•   Full Text pdf for Viewing (14 MB) - http://www.unicode.org/versions/Unicode12.0.0/UnicodeStandard-12.0.pdf

Coded Character Sets, History and Development :
https://archive.org/details/mackenzie-coded-char-sets
https://archive.org/download/mackenzie-coded-char-sets/Mackenzie_CodedCharSets.pdf

Writing Unicode programs - GoAsm manual - http://www.godevtool.com/GoasmHelp/Unicode.htm

Main part

Why Non-Unicode apps system locale makes Unicode fonts with symbol charset displayed incorrectly? - https://stackoverflow.com/questions/21383099/why-non-unicode-apps-system-locale-makes-unicode-fonts-with-symbol-charset-displ

The root cause of the problem is that Wingdings font is actually non-Unicode font. It supports Unicode partially, so some symbols are still displayed correctly. See @Adrian McCarthy's answer for details about how it's probably works under the hood.

Also see more info here: http://www.fileformat.info/info/unicode/font/wingdings and here: http://www.alanwood.net/demos/wingdings.html

So what can we do to avoid such problems? I found several ways:

1. Quick & dirty

Fall back to ANSI version of API, as @user1793036 suggested:

Code: [Select]

TextOutA(hdc, 10, 10, "\x7d\x7e\x81\xfc"); // Displayed correctly!
2. Quick & clean

Use special Unicode range F0 (Private Use Area - https://en.wikipedia.org/wiki/Private_Use_Areas) instead of ASCII character codes. It's supported by Wingdings:

Code: [Select]

TextOutW(hdc, 10, 10, L"\xf07d\xf07e\xf081\xf0fc"); // Displayed correctly!To explore which Unicode symbols are actually supported by font some font viewer can be used, e.g. dp4 Font Viewer

3. Slow & clean, but generic

But what to do if you don't know which characters you have to display and which font actually will be used? Here is most universal solution - draw text by glyphs to avoid any undesired translations:

Code: [Select]

void TextOutByGlyphs(HDC hdc, int x, int y, const CStringW& text)
{
   CStringW glyphs;

   GCP_RESULTSW gcpRes = {0};
   gcpRes.lStructSize = sizeof(GCP_RESULTS);
   gcpRes.lpGlyphs = glyphs.GetBuffer(text.GetLength());
   gcpRes.nGlyphs = text.GetLength();

   const DWORD flags = GetFontLanguageInfo(hdc) & FLI_MASK;
   GetCharacterPlacementW(hdc, text.GetString(), text.GetLength(), 0,
     &gcpRes, flags);
   glyphs.ReleaseBuffer(gcpRes.nGlyphs);

   ExtTextOutW(hdc, x, y, ETO_GLYPH_INDEX, NULL, glyphs.GetString(),
      glyphs.GetLength(), NULL);
}

Code: [Select]

TextOutByGlyphs(hdc, 10, 10, L"\x7d\x7e\x81\xfc"); // Displayed correctly!Note GetCharacterPlacementW() function usage. For some unknown reason similar function GetGlyphIndicesW() would not work returning 'unsupported' dummy values for chars > 127.


@Adrian McCarthy's

Here's what I think is happening:

The Wingdings font doesn't have Unicode mappings (a cmap table?). (You can see this by using charmap.exe: the Character set drop down control is grayed out.)
For fonts without Unicode mappings, I think Windows assumes that it depends on the "Language for Non-Unicode applications" setting.
When that's English, Windows (probably) uses code page 1252, and all the values map to themselves.
When that's Russian, Windows (probably) uses code page 1251, and then tries to remap them.
The '\x81' value in code page 1251 maps to U+0403, which obviously doesn't exist in the font, so you get a box. Similarly the, '\xFC' maps to U+044C.
I assumed that if you used ExtTextOutW with the ETO_GLYPH_INDEX flag, Windows wouldn't try to interpret the values at all and just treat them as glyph indexes into the font. But that assumption is wrong.
However, there is another flag called ETO_IGNORELANGUAGE, which is reserved, but, empirically, it seems to solve the problem.

user1793036
I recall that on a Unicode build of my app some symbols would get the rectangle glyph, where in the Ansi build they would display fine. When I searched online I discovered that WingDings is only partially supported in Unicode. My solution was to draw each character using DrawTextA. This worked for me because I was only drawing individual characters.
Rost
Unfortunately ETO_IGNORELANGUAGE is not working, still same boxes are drawn.

[Biterider]

Hi LiaoMi
Thank you for pointing out this article. It has a lot of useful information!
According to the reasoning, the reason your system is responding differently may be due to the system's locale. So let's compare it. My is "Deutsch(Schweiz)" 

It seems that the reserved area (2. Quick & clean) is that what I found for these fonts.

Biterider

[LiaoMi]

Hi LiaoMi
Thank you for pointing out this article. It has a lot of useful information!
According to the reasoning, the reason your system is responding differently may be due to the system's locale. So let's compare it. My is "Deutsch(Schweiz)" 

It seems that the reserved area (2. Quick & clean) is that what I found for these fonts.

Biterider

Hi Biterider,

most likely it is true, https://i.ibb.co/vJBH7Qm/2021-05-15-10-48-45.jpg and https://i.ibb.co/SVCRZyx/2021-05-15-11-44-08-Region.png , https://i.ibb.co/5M1jJsX/2021-05-15-11-52-30-Text-View.jpg.

Yes, the second solution is like yours 

[Biterider]

Hi LiaoMi
I tried your system configuration but could not reproduce your character mapping.
Using Windows in a language I don't know was a strange experience. I found most of the system interactions just looking at the icons 

Biterider

[LiaoMi]

Hi LiaoMi
I tried your system configuration but could not reproduce your character mapping.
Using Windows in a language I don't know was a strange experience. I found most of the system interactions just looking at the icons 

Biterider

  I also did not succeed in repeating it with your configuration. Perhaps this does not work without a reboot, or these parameters are permanently saved to the registry (without reinstalling the system will not work?)