Author Topic: TextView  (Read 5551 times)

fearless

  • Member
  • ****
  • Posts: 515
    • Github
Re: TextView
« Reply #30 on: March 22, 2021, 12:43:36 AM »
Also found out about this Private Character Editor (eudcedit.exe) which comes with every version of windows: https://www.makeuseof.com/tag/create-your-own-fonts-symbols-characters-windows/
fearless

Lian Li PC-O11DW Case, ASUS ROG Crosshair VIII Hero WiFi, AMD Ryzen 9 5950X, 32GB DDR4-3600MHz, MSI RX 5700XT Gaming X, NZXT Kraken Z73 360mm Cooler, Seasonic PRIME 1000W PSU

My Github  Twitter

Biterider

  • Moderator
  • Member
  • *****
  • Posts: 723
  • ObjAsm Developer
    • ObjAsm
Re: TextView
« Reply #31 on: March 22, 2021, 01:17:13 AM »
Hi
I have never heard of this "Private Character Editor" before.
Incredible that it has always been there.  :biggrin:

Biterider

Biterider

  • Moderator
  • Member
  • *****
  • Posts: 723
  • ObjAsm Developer
    • ObjAsm
Re: TextView
« Reply #32 on: March 22, 2021, 05:08:18 AM »
Hi
Searching for more information I found a useful and free tool to visualize the Unicode mapping.
https://www.babelstone.co.uk/Software/BabelMap.html

Biterider

Biterider

  • Moderator
  • Member
  • *****
  • Posts: 723
  • ObjAsm Developer
    • ObjAsm
Re: TextView
« Reply #33 on: May 14, 2021, 02:15:42 AM »
Hi
I spent some time looking for a solution to the above issue.
Since nothing really worked and there was no useful information on the internet, I tried to find out for myself.

I use the DrawText API and the "Wingding 2" font to display some symbol characters in the TextView control. In particular, I was irritated to see missing glyphs near character 128 and above (empty square). My hope is that if I can solve this problem, I will have the solution for Liaomi's system too.

I started experimenting with the ExtTextOut API. It has a very useful flag called ETO_GLYPH_INDEX that I used to output all the characters in the font in sequence. To my surprise, all of the glyphs we can see in the "font insertion dialogbox" from a MS application were there, even when DrawText refuses to display them.
I suspected that DrawText was doing some sort of mapping when using some particular fonts.  :icon_idea:
I wrote a short program to show all characters wider than a predefined threshold and in the range 0000 to FFFFh and voilá. All the characters were there, but in very different positions!
I noticed that in the range from F000h to F0FFh all characters are in the correct order. That said, when we "and F000h" to the character value, we get the correct glyph from DrawText.

I don't have an explanation why this mapping is happening and I don't see any correlation with the Unicode BMP.  :sad:

This analysis is only valid for some sort of fonts. Other symbol fonts like Symbol, Webdings, Wingding, Wingding 3, behave the same way, while other regular fonts do not.

Changing the the charset to SYMBOL_CHARSET doesn't have any effect on the mapping.

If anyone has a clue on this topic, please post it here.

I changed the demo application to take this behavior into account. Additionally I show the characters in the range 0020h-00FFh and F020-F0FFh.
I was intrigued to see if the problem was gone on Liaomi's PC.

Biterider


jj2007

  • Member
  • *****
  • Posts: 11550
  • Assembler is fun ;-)
    • MasmBasic
Re: TextView
« Reply #34 on: May 14, 2021, 02:51:09 AM »
Win7-64

LiaoMi

  • Member
  • ****
  • Posts: 922
Re: TextView
« Reply #35 on: May 14, 2021, 06:30:19 AM »
Hi
I spent some time looking for a solution to the above issue.
Since nothing really worked and there was no useful information on the internet, I tried to find out for myself.

I use the DrawText API and the "Wingding 2" font to display some symbol characters in the TextView control. In particular, I was irritated to see missing glyphs near character 128 and above (empty square). My hope is that if I can solve this problem, I will have the solution for Liaomi's system too.

I started experimenting with the ExtTextOut API. It has a very useful flag called ETO_GLYPH_INDEX that I used to output all the characters in the font in sequence. To my surprise, all of the glyphs we can see in the "font insertion dialogbox" from a MS application were there, even when DrawText refuses to display them.
I suspected that DrawText was doing some sort of mapping when using some particular fonts.  :icon_idea:
I wrote a short program to show all characters wider than a predefined threshold and in the range 0000 to FFFFh and voilá. All the characters were there, but in very different positions!
I noticed that in the range from F000h to F0FFh all characters are in the correct order. That said, when we "and F000h" to the character value, we get the correct glyph from DrawText.

I don't have an explanation why this mapping is happening and I don't see any correlation with the Unicode BMP.  :sad:

This analysis is only valid for some sort of fonts. Other symbol fonts like Symbol, Webdings, Wingding, Wingding 3, behave the same way, while other regular fonts do not.

Changing the the charset to SYMBOL_CHARSET doesn't have any effect on the mapping.

If anyone has a clue on this topic, please post it here.

I changed the demo application to take this behavior into account. Additionally I show the characters in the range 0020h-00FFh and F020-F0FFh.
I was intrigued to see if the problem was gone on Liaomi's PC.

Biterider

Hi Biterider,

very cool, great job! You can write an article, because even in windows there is a similar glitch with the display of squares. The picture shows that everything works fine. But I could not increase the height of the window  :biggrin:



@jj2007 - did you manually change the height of the window?  :tongue:
P.S> Ah, now I see you glued two pictures together  :thup: Your test also has problems with displaying the font. But in the modification, all these problems have now been eliminated.

FORTRANS

  • Member
  • *****
  • Posts: 1109
Re: TextView
« Reply #36 on: May 14, 2021, 06:50:33 AM »
Hi,

   Windows 10.  Cropped screenshot.

Steve

jj2007

  • Member
  • *****
  • Posts: 11550
  • Assembler is fun ;-)
    • MasmBasic
Re: TextView
« Reply #37 on: May 14, 2021, 07:03:29 AM »
@jj2007 - did you manually change the height of the window?  :tongue:
P.S> Ah, now I see you glued two pictures together  :thup: Your test also has problems with displaying the font. But in the modification, all these problems have now been eliminated.

PaintShop 4.1 is my friend :tongue:

nidud

  • Member
  • *****
  • Posts: 2213
    • https://github.com/nidud/asmc
Re: TextView
« Reply #38 on: May 14, 2021, 07:22:45 AM »
Not sure this relate but I made a TrueType font once using FontForge for TextView to display ASCII symbols (viewing binary files).

http://masm32.com/board/index.php?topic=2344.msg26000#msg26000

Biterider

  • Moderator
  • Member
  • *****
  • Posts: 723
  • ObjAsm Developer
    • ObjAsm
Re: TextView
« Reply #39 on: May 14, 2021, 03:18:46 PM »
Hi
Thank you for testing. It seems to work fine, but it remains a mystery why Liaomi's system works differently.  :rolleyes:

@nidud: thanks, I'll check it.

Biterider

LiaoMi

  • Member
  • ****
  • Posts: 922
Re: TextView
« Reply #40 on: May 14, 2021, 06:01:39 PM »
but it remains a mystery why Liaomi's system works differently.  :rolleyes:

Hi Biterider,

Prologue>

Version 12.0.0 has been superseded by the latest version of the Unicode Standard - https://unicode.org/versions/Unicode12.0.0/
All Chapters and Appendices Together:
•   Full Text pdf for Viewing (14 MB) - http://www.unicode.org/versions/Unicode12.0.0/UnicodeStandard-12.0.pdf

Coded Character Sets, History and Development :
https://archive.org/details/mackenzie-coded-char-sets
https://archive.org/download/mackenzie-coded-char-sets/Mackenzie_CodedCharSets.pdf

Writing Unicode programs - GoAsm manual - http://www.godevtool.com/GoasmHelp/Unicode.htm

Main part

Why Non-Unicode apps system locale makes Unicode fonts with symbol charset displayed incorrectly? - https://stackoverflow.com/questions/21383099/why-non-unicode-apps-system-locale-makes-unicode-fonts-with-symbol-charset-displ

The root cause of the problem is that Wingdings font is actually non-Unicode font. It supports Unicode partially, so some symbols are still displayed correctly. See @Adrian McCarthy's answer for details about how it's probably works under the hood.

Also see more info here: http://www.fileformat.info/info/unicode/font/wingdings and here: http://www.alanwood.net/demos/wingdings.html

So what can we do to avoid such problems? I found several ways:

1. Quick & dirty

Fall back to ANSI version of API, as @user1793036 suggested:

Code: [Select]
TextOutA(hdc, 10, 10, "\x7d\x7e\x81\xfc"); // Displayed correctly!
2. Quick & clean

Use special Unicode range F0 (Private Use Area - https://en.wikipedia.org/wiki/Private_Use_Areas) instead of ASCII character codes. It's supported by Wingdings:

Code: [Select]
TextOutW(hdc, 10, 10, L"\xf07d\xf07e\xf081\xf0fc"); // Displayed correctly!To explore which Unicode symbols are actually supported by font some font viewer can be used, e.g. dp4 Font Viewer

3. Slow & clean, but generic

But what to do if you don't know which characters you have to display and which font actually will be used? Here is most universal solution - draw text by glyphs to avoid any undesired translations:

Code: [Select]
void TextOutByGlyphs(HDC hdc, int x, int y, const CStringW& text)
{
   CStringW glyphs;

   GCP_RESULTSW gcpRes = {0};
   gcpRes.lStructSize = sizeof(GCP_RESULTS);
   gcpRes.lpGlyphs = glyphs.GetBuffer(text.GetLength());
   gcpRes.nGlyphs = text.GetLength();

   const DWORD flags = GetFontLanguageInfo(hdc) & FLI_MASK;
   GetCharacterPlacementW(hdc, text.GetString(), text.GetLength(), 0,
     &gcpRes, flags);
   glyphs.ReleaseBuffer(gcpRes.nGlyphs);

   ExtTextOutW(hdc, x, y, ETO_GLYPH_INDEX, NULL, glyphs.GetString(),
      glyphs.GetLength(), NULL);
}

Code: [Select]
TextOutByGlyphs(hdc, 10, 10, L"\x7d\x7e\x81\xfc"); // Displayed correctly!Note GetCharacterPlacementW() function usage. For some unknown reason similar function GetGlyphIndicesW() would not work returning 'unsupported' dummy values for chars > 127.


@Adrian McCarthy's

Here's what I think is happening:

The Wingdings font doesn't have Unicode mappings (a cmap table?). (You can see this by using charmap.exe: the Character set drop down control is grayed out.)
For fonts without Unicode mappings, I think Windows assumes that it depends on the "Language for Non-Unicode applications" setting.
When that's English, Windows (probably) uses code page 1252, and all the values map to themselves.
When that's Russian, Windows (probably) uses code page 1251, and then tries to remap them.
The '\x81' value in code page 1251 maps to U+0403, which obviously doesn't exist in the font, so you get a box. Similarly the, '\xFC' maps to U+044C.
I assumed that if you used ExtTextOutW with the ETO_GLYPH_INDEX flag, Windows wouldn't try to interpret the values at all and just treat them as glyph indexes into the font. But that assumption is wrong.
However, there is another flag called ETO_IGNORELANGUAGE, which is reserved, but, empirically, it seems to solve the problem.

user1793036
I recall that on a Unicode build of my app some symbols would get the rectangle glyph, where in the Ansi build they would display fine. When I searched online I discovered that WingDings is only partially supported in Unicode. My solution was to draw each character using DrawTextA. This worked for me because I was only drawing individual characters.
Rost
Unfortunately ETO_IGNORELANGUAGE is not working, still same boxes are drawn.

Biterider

  • Moderator
  • Member
  • *****
  • Posts: 723
  • ObjAsm Developer
    • ObjAsm
Re: TextView
« Reply #41 on: May 14, 2021, 10:57:06 PM »
Hi LiaoMi
Thank you for pointing out this article. It has a lot of useful information!
According to the reasoning, the reason your system is responding differently may be due to the system's locale. So let's compare it. My is "Deutsch(Schweiz)"  :biggrin:

It seems that the reserved area (2. Quick & clean) is that what I found for these fonts.

Biterider
« Last Edit: May 15, 2021, 03:12:25 PM by Biterider »

LiaoMi

  • Member
  • ****
  • Posts: 922
Re: TextView
« Reply #42 on: May 15, 2021, 07:51:22 PM »
Hi LiaoMi
Thank you for pointing out this article. It has a lot of useful information!
According to the reasoning, the reason your system is responding differently may be due to the system's locale. So let's compare it. My is "Deutsch(Schweiz)"  :biggrin:

It seems that the reserved area (2. Quick & clean) is that what I found for these fonts.

Biterider

Hi Biterider,

most likely it is true, https://i.ibb.co/vJBH7Qm/2021-05-15-10-48-45.jpg and https://i.ibb.co/SVCRZyx/2021-05-15-11-44-08-Region.png , https://i.ibb.co/5M1jJsX/2021-05-15-11-52-30-Text-View.jpg.

Yes, the second solution is like yours  :thumbsup:

Biterider

  • Moderator
  • Member
  • *****
  • Posts: 723
  • ObjAsm Developer
    • ObjAsm
Re: TextView
« Reply #43 on: May 15, 2021, 09:13:59 PM »
Hi LiaoMi
I tried your system configuration but could not reproduce your character mapping.
Using Windows in a language I don't know was a strange experience. I found most of the system interactions just looking at the icons  :tongue:

Biterider

LiaoMi

  • Member
  • ****
  • Posts: 922
Re: TextView
« Reply #44 on: May 15, 2021, 09:32:33 PM »
Hi LiaoMi
I tried your system configuration but could not reproduce your character mapping.
Using Windows in a language I don't know was a strange experience. I found most of the system interactions just looking at the icons  :tongue:

Biterider

 :biggrin: I also did not succeed in repeating it with your configuration. Perhaps this does not work without a reboot, or these parameters are permanently saved to the registry (without reinstalling the system will not work?)