The MASM Forum

Projects => ObjAsm => Topic started by: Biterider on May 07, 2020, 06:34:09 PM

Title: TextView
Post by: Biterider on May 07, 2020, 06:34:09 PM
Hi
While developing an application, I had to display formatted text and used the RichEdit control for this purpose. I found it to be an overkill and DLL nightmare, so I decided to look for alternatives.

I analysed the amazing and well known works of:
and decided to go a somewhat different way. Instead of using HTML, I created my own markup definition. The goal was that it should be simple, MASM, keyboard and printf friendly.

I experimented with different syntaxes and finally found two sets of tags: styles and commands.
A style defines how text is displayed (font, size, color, background, etc.), while commands trigger immediate actions (tab stops, line breaks, paragraphs, etc.). As with the previous implementations, styles are managed in a stack, so that previous sytles can be reused as soon as the newer style is popped off.
TextView is created in the form of a control to facilitate the implementation of active areas. These areas are sensitive to user interactions such as hyperlinks, buttons, tooltips, etc.
Here is a summary of the currently implemented markups:

Styles:
APP   Application notification
SUB   Subscript
SUP   Superscript
BOLD   Bold
FONT   Set font face
INFO   Information callback
ALIGN   Set alignment
COLOR   Set background color
HEIGHT   Set font height
INDENT   Set indentation
ITALIC   Italic
BGCOLOR   Set background color
STRIKEOUT   Strikeout
UNDERLINE   Underline


Commands:
TAB   Performs a tabstop
DRAW   Parent draw
PARA   Start a new paragraph
TABC   Define a "tabstop center"
TABL   Define a "Tabstop left"
TABR   Define a "Tabstop right"
BREAK   Perform a line break
COVER   Parent drawing covering all previous actions
SIZE.X   Set draw area width
SIZE.Y   Set draw area height
TABCLR   Reset all tabstops
PADDING   All paddings
PADDING.TOP   Set pop padding
PADDING.LEFT   Set left padding
PADDING.RIGHT   Set right padding
PADDING.BOTTOM   Set bottom padding


The drawing process takes place in two steps. The first step is to parse the markup string using a state machine and generate all the GDI style resources (fonts, brushes, etc.). This output is used in the second step to execute the commands and render the individual drawing units.

There are three parent callbacks implemented. APP, INFO and DRAWING. The APP callback informs the parent window about mouse activities in the active areas. INFO callbacks request information to display tool tips, and finally DRAW callbacks request drawing operations. This last type of callback can be executed during the rendering process or at the end, covering all previous rendering tasks.

I attach two executables showing the TextView replacement of the standard about dialog (check the active areas - circled i) and a demo showing the tabbing, indenting and sub- and super-scripting capabilities.

Here ist the source of the markup string of this last Demo:

CStr szMarkupText,    "{Padding 20}", \
                      "{TabC 130}{TabR 210}", \
                      "[Indent 10, H +2, Bold, Underline]", \
                        "Description{Tab}Qty{TAB}Price{P}", \
                      "[~]", \
                      "[Indent 15, Color #4F4FFF, H +2, Italic]", \
                          "{#2022} Item[Sub]1[~]{Tab}2{Tab}2.35[Sup]*[~]{B}", \
                          "{#2022} Item[Sub]2[~]{Tab}200.0{Tab}20.00[Sup]**[~]{B}", \
                          "{#2022} Item[Sub]3[~]{Tab}0.1{Tab}0.23{B}{P}", \
                      "[~]", \
                      "{TabClr}{TabC 80}{TabL 95}", \
                      "[Indent 30, Color #4F4F4F]", \
                        "Note:{Tab}*{Tab}Volume discount{B}", \
                        "{Tab}**{Tab}Regular discount", \
                      "[~]", \


The source code will be released soon in the next ObjAsm update.

Have fun,  :biggrin:

Biterider
Title: Re: TextView
Post by: jj2007 on May 08, 2020, 12:25:24 AM
Nice :thumbsup: When does the Rtf-TV converter come?
Title: Re: TextView
Post by: Biterider on May 08, 2020, 01:50:46 AM
Hi JJ
No plan for such a converter, but for a WYSIWYG editor, like the one of Hans Dietrich.
In that way, you see immediately the markup effect while typing it.

Biterider
Title: Re: TextView
Post by: HSE on May 09, 2020, 12:23:10 AM
Hi Biterider!

Quote from: Biterider on May 07, 2020, 06:34:09 PM
While developing an application, I had to display formatted text and used the RichEdit control for this purpose. I found it to be an overkill and DLL nightmare, so I decided to look for alternatives.

Fantastic idea!

We have seen how You use formatted text. And so far TextViewer look pretty good for that. I think to add paragraph center option could improve the control a lot. Of course I'm thinking how I can use the control, but here table example only use left half of the window, and don't look so impressive. About example, on contrary, is in right and bottom limits.

Regards. HSE
Title: Re: TextView
Post by: Biterider on May 09, 2020, 02:46:19 AM
Hi HSE
This is the markup string for the About Dialog

"{Padding 20}", \
"[Height 11]", \
  "{Cover 0}", \
  "[Indent 150, Align 'LEFT']", \
    "[Height +3, Bold]&ABOUT_TEXT{B}[~]", \
    "[Italic]Version &VER_PRODUCTVERSION_STR{P}[~]", \
    "[Color #3F3F3F]", \
      "Designed with ObjAsm{B}", \
      "© &COPYRIGHT{P}", \
      "[Color #5F4FFF]", \
        @CatStr(<!">, %BUILD_DATE_STR, \
                <  -  Build: >, %BUILD_NUMBER, <->, %ASSEMBLER, <->, \
                %TARGET_MODE_STR, <->, %TARGET_BITNESS, <    !">), \
        "[Info ", @CatStr(<!">, %INFO_BUILD, <!">), ", Font 'Webdings', Height +1]{#69}[~]", \
      "[~]{B}{B}", \
    "[~]", \
  "[~]", \
  "[Indent 0, Align 'CENTER']", \
    "[Font 'Webdings', Color #FF5F5F, Height +7]{#7E}[~]", \
    "  Visit us at  ", \
    "[App ", @CatStr(<!">, %APP_VISIT, <!">), ", Color #0000FF, U]www.objasm.x10host.com[~]{B}", \
    "[Font 'Wingdings', Color #408090, Height +1]{#2A}[~]", \
    "   Email us to  ", \
    "[App ", @CatStr(<!">, %APP_EMAIL, <!">), ", Color #0000FF, U]objasm@gmx.net[~]{P}", \
    "[App ", @CatStr(<!">, %APP_CHECK, <!">), ", Color #0000FF, U]Check for updates[~]", \
    "[Info ", @CatStr(<!">, %INFO_UPDATE, <!">), ", Font 'Webdings', H +1, Color #5F5FFF]  {#69}[~]", \
  "[~]", \
"[~]"


I used @CatStr(...) to embed some constants I used on the parent window

APP_VISIT     equ   0
APP_EMAIL     equ   1
APP_CHECK     equ   2
INFO_BUILD    equ   0
INFO_UPDATE   equ   1


The implementation is pretty straight forward

Method Application.OnNotify, uses xdi xsi, wParam:WPARAM, lParam:LPARAM
  local Img:$Obj(Image), DstRect:RECT

  SetObject xsi
  mov xax, wParam
  .if xax == VIEW_CTRL_ID
    mov xdi, lParam
    .if [xdi].NMHDR.code == TVNMC_APP_MOUSEUP
      .if [xdi].TVNM_APP.dID == APP_VISIT
        invoke ShellExecute, [xsi].hWnd, $OfsCStr("open"), \
                             $OfsCStr("http:/objasm.x10host.com/index.htm"), \
                             NULL, NULL, SW_SHOW
      .elseif [xdi].TVNM_APP.dID == APP_EMAIL
        invoke ShellExecute, [xsi].hWnd, $OfsCStr("open"), \
                             $OfsCStr("mailto:objasm@gmx.net"), \
                             NULL, NULL, SW_SHOW
      .elseif [xdi].TVNM_APP.dID == APP_CHECK
        invoke MessageBox, [xsi].hWnd, $OfsCStr("No update available."), \
                                       $OfsCStr("Information"), \
                                       MB_ICONINFORMATION or MB_OK
      .endif

    .elseif [xdi].NMHDR.code == TVNMC_INFO
      .if [xdi].TVNM_INFO.dID == INFO_BUILD
        c2m [xdi].TVNM_INFO.pText, offset szBuildInfo, xax

      .elseif [xdi].TVNM_INFO.dID == INFO_UPDATE
        c2m [xdi].TVNM_INFO.pText, offset szUpdateInfo, xax
      .endif

    .elseif [xdi].NMHDR.code == TVNMC_DRAW
      .if [xdi].TVNM_DRAW.dID == 0
        New Img::Image
        OCall Img::Image.Init, xsi
        OCall Img::Image.%LoadFromRes, $OfsCStr("OA")
        mov DstRect.left, 35
        mov DstRect.top, 35
        mov DstRect.right, 100
        mov DstRect.bottom, 100
        OCall Img::Image.AlphaDraw, [xdi].TVNM_DRAW.hDC, 255, addr DstRect
        OCall Img::Image.Done
      .endif
    .endif
  .endif
MethodEnd


Biterider
Title: Re: TextView
Post by: HSE on May 09, 2020, 03:24:15 AM
Ok  :thumbsup:

We have to test that  :biggrin:

Thanks
Title: Re: TextView
Post by: Biterider on May 13, 2020, 04:01:27 AM
Hi
A little bit of color makes it look a lot better.  :biggrin:
This demo shows text alignment to the left/center/right using tabs, subscript and superscript, indentation, different font styles and sizes, colors and PNG shrinking and alphablending.

Biterider
Title: Re: TextView
Post by: daydreamer on May 30, 2020, 11:37:44 PM
so it would be possible to split first part parsing to a "compiler" part and the program you release have that in smaller "compiled" data
Title: Re: TextView
Post by: Biterider on May 31, 2020, 01:25:48 AM
Hi
Yes, technically possible. The other way is to use the buildin streaming capabilities to save the content of the collections into a stream and store these data afterward into the resource section. A bit more work has to be done, but it is definitively possible.

Biterider
Title: Re: TextView
Post by: daydreamer on May 31, 2020, 08:10:14 AM
Quote from: Biterider on May 31, 2020, 01:25:48 AM
Hi
Yes, technically possible. The other way is to use the buildin streaming capabilities to save the content of the collections into a stream and store these data afterward into the resource section. A bit more work has to be done, but it is definitively possible.

Biterider
sounds good
I have recently worked with HTML/scripting and learned some things
maybe accompany it later with a event handler proc's that or several procs that does interactive things similar that scripts in HTML does?
Title: Re: TextView
Post by: Biterider on May 31, 2020, 02:59:58 PM
Hi daydreamer
Similar behavior has already been implemented. If you look closely, you'll see a collection called ActiveTVAreas. It contains the "hot" areas that can interact with the user.
In the current implementation, when the mouse interacts with the "hot" areas, information is sent or requested from the parent window using notifications. This is how the hyperlink and tooltip work.
I can see many other ways to use it too, e.g. to create a button or more complex controls.
Still, I didn't go any further because I thought it would be a better approach to use the traditional controls to keep functionality and complexity where they belong.

Biterider
Title: Re: TextView
Post by: Biterider on June 15, 2020, 06:52:29 AM
Hi
I have completed the last development step of this control. It now comes very close to the amazing work of fearless :cool:
http://masm32.com/board/index.php?topic=7372.0 (http://masm32.com/board/index.php?topic=7372.0)

I added more functionality, word wrapping, text boxing, scrollbars and mouse wheel support and cleaned up the code.

Biterider
Title: Re: TextView
Post by: Biterider on March 06, 2021, 10:25:56 PM
Hi
Today I want to show you the new TextView. It's a completely redesigned version where I've implemented what I learned from the previous code. In addition to heavy refactoring, I added some very useful features.

The aim of TextView is to draw reach-formatted text easily and fully automatically, taking over the management of all resources required for the drawing process. It uses a markup text to instruct what to do and set the styles for the drawn text. This code is completely GDI based. This allows you to render on any device that supports Windows, even without GPU support.

This version was developed for the usual 32/64 and ANSI/WIDE targets. Internally, however, TextView works with WIDE strings, enabling the rendering of all characters that Windows can draw (BMP). If you compile for ANSI string target or you can't enter Unicode characters in your editor, you can enter them using dedicated markups. For example, this situation may occur when you want to list bullets.

Zooming is one of the added new features. It is vector-based and allows you to draw scaled text without any loss of accuracy.
To enable use on mobile devices, gesture support for panning, over-panning and zooming is also supported beyond the legacy windows implementation.
All the rendering options are global and can now be set in the markup string too.

With the "Automatic line break" option, you can adapt automatically the drawing to different client sizes. A nice example are message-boxes with large texts. They adapt to a dynamic layout or display a vertical scrollbar to see the rendered text on a smaller view.

Supported functions are also styles, short markups, tooltips, active areas, adjustable tabs, line breaks, paragraphs, indentations with hanging, custom padding on all edges, text and graphic callbacks, text boxes, alignment, etc.

In the background, the rendering process uses double buffering, which results in flicker-free visualization. Due to the use of look-up tables, the analysis, pre-calculation and drawing process is very fast. On my system, the representation of a markup string of about 100 commands was parsed and precomputed in 25 microseconds and rendered in 5 milliseconds.

TextView is lightweight. It has a code footprint of 30 KB in 64 bit.

As usual, it is licence free and open source.

Attached are 2 revised examples. One shows the rendering features, active areas and tooltips, while the second shows zooming and gesture handling.
The documentation is included in the source code file on GitHub.

Have fun.  :cool:

Biterider
Title: Re: TextView
Post by: TouEnMasm on March 07, 2021, 01:31:54 AM

creating a control edit is a good idea.
Nightmare come with Msftedit , need to be used only in unicode or to have big problems.
If TextView can do the same thing than a richedit,that will be perfect.
Title: Re: TextView
Post by: fearless on March 07, 2021, 02:00:37 AM
Excellent stuff. Its looking great.
Title: Re: TextView
Post by: Biterider on March 07, 2021, 09:13:43 PM
Hi
Well, I have to clarify a misunderstanding.  :cool:
RichEdit is basically an edit control that is able to interact with the user and display rich-content.
TextView, on the other hand, was developed to display rich-content in a simple way using markups and handle the resource management automatically. It is not designed as an input control. The lack of this functionality has the great advantage that it is very light and fast. When text entry is required, a classic edit control can do the job.

Biterider
Title: TextView Controls
Post by: Biterider on March 13, 2021, 06:05:26 PM
Hi
This is a TextView implementation of simple zoomable controls.

The technique used is a combination of active areas and text callbacks. The latter was implemented in the most recent incarnation of this component.
When a mouse or touch interaction is detected in an active area, the implementation of the control responds by setting the appropriate text using the text callback. In the case of the radio button or check box, Unicode characters of the same width have been used to leave the rest of the text unchanged. This way, no recalculation is required, which saves processing time. The only requirement is that the fonts used are installed on the host system. I used "Lucida Sans Unicode" and "Wingdings" which should be included with every Windows installation.

This is a solution for situations where simple interaction is required in a text-based context.
The bonus point is that since it's text based, it's also zoomable, as seen in the attached demo.

Have fun!

Biterider
Title: Re: TextView Controls
Post by: LiaoMi on March 19, 2021, 06:00:49 AM
Quote from: Biterider on March 13, 2021, 06:05:26 PM
Hi
This is a TextView implementation of simple zoomable controls.

The technique used is a combination of active areas and text callbacks. The latter was implemented in the most recent incarnation of this component.
When a mouse or touch interaction is detected in an active area, the implementation of the control responds by setting the appropriate text using the text callback. In the case of the radio button or check box, Unicode characters of the same width have been used to leave the rest of the text unchanged. This way, no recalculation is required, which saves processing time. The only requirement is that the fonts used are installed on the host system. I used "Lucida Sans Unicode" and "Wingdings" which should be included with every Windows installation.

This is a solution for situations where simple interaction is required in a text-based context.
The bonus point is that since it's text based, it's also zoomable, as seen in the attached demo.

Have fun!

Biterider

Hi Biterider,

I have some elements displayed incorrectly :rolleyes:
(https://i.ibb.co/nRxpMDy/2021-03-18-19-44-19-Text-View-Demo.png)
Title: Re: TextView
Post by: Biterider on March 19, 2021, 08:42:50 AM
Hi LiaoMi
Thanks for your feedback. I have a guess as to what is happening here. The selected characters for the checkboxes are 00A3 and 0052 of "Wingdings 2" font. The former (unchecked checkbox) is used to calculate character width. I see on the "Option 4" line a character that is used when the rendering fails. This smaller character reduces the space for the checked checkbox and that is what we see.

Can you check that you have this font installed and what these characters look like when using for instance the option "insert symbols" of MS-Word?
To find out what happens, I can prepare a binary to display the renderable characters of that font.

Thank you for your support.

Biterider
Title: Re: TextView
Post by: LiaoMi on March 19, 2021, 08:52:35 AM
Quote from: Biterider on March 19, 2021, 08:42:50 AM
Hi LiaoMi
Thanks for your feedback. I have a guess as to what is happening here. The selected characters for the checkboxes are 00A3 and 0052 of "Wingdings 2" font. The former (unchecked checkbox) is used to calculate character width. I see on the "Option 4" line a character that is used when the rendering fails. This smaller character reduces the space for the checked checkbox and that is what we see.

Can you check that you have this font installed and what these characters look like when using for instance the option "insert symbols" of MS-Word?
To find out what happens, I can prepare a binary to display the renderable characters of that font.

Thank you for your support.

Biterider

The font is installed on the system and looks like
(https://i.ibb.co/LzPYPmq/2021-03-18-22-50-11-Document1-Word.png)
Title: Re: TextView
Post by: Biterider on March 20, 2021, 07:26:14 PM
Hi LiaoMi
I couldn't find an obvious bug on my side, so I decided to add some debug outputs into the code and redirect them to the console (in case you don't have DebugCenter installed).
I also added all characters of the "Wingdings 2" font to the TextView output and found that not all of them are rendered as they should (compared with MS-Mord) and a square is rendered in place of the expected character.

I am very interested to see how this looks on your system.

Biterider
Title: Re: TextView
Post by: LiaoMi on March 20, 2021, 10:23:17 PM
Hi Biterider,

Debug view  :arrow_left:
(https://i.ibb.co/P6C9FLL/2021-03-20-12-18-36-Text-View-Demo.jpg)
Title: Re: TextView
Post by: Biterider on March 20, 2021, 10:39:00 PM
Hi LiaoMi
That is a surprising result. The debug output confirms that the characters reaching the DrawText API are correct.
On my system I can render almost all characters from this font, but on your system it is very different. About half of it cannot be rendered. I've searched the internet and found some evidence of character mapping errors on OS level, but I can't understand how it relates to this effect..
Which OS are you using?

Biterider
Title: Re: TextView
Post by: LiaoMi on March 20, 2021, 11:05:03 PM
@Biterider

QuoteWhich OS are you using?

Windows 10 Enterprise Version 1909 Build 18363.418

A couple more links..
http://www.alanwood.net/demos/wingdings-2.html
https://stackoverflow.com/questions/22092370/windows-form-controls-font-not-changing-for-wingding

Other examples also do not display the font correctly.
(https://i.ibb.co/XfQRntD/2021-03-20-13-00-10-Document1-Word.jpg)
Title: Re: TextView
Post by: LiaoMi on March 20, 2021, 11:29:39 PM
(https://i.ibb.co/t3ZXpG1/2021-03-20-13-25-04-Settings.png)
and
(https://i.ibb.co/QdSC28W/2021-03-20-13-32-18-Settings2.png)

But I'm not sure if this is the case here:
Why does some text display with square boxes in some apps on Windows 10?
https://support.microsoft.com/en-us/topic/why-does-some-text-display-with-square-boxes-in-some-apps-on-windows-10-b078a35f-9709-1780-44c0-8c27a58205a2

A fallback font is a font which will be used if your other choices aren't usable
https://github.com/gillesgg/DxFontFallback
https://github.com/gillesgg/DxFontFallback/archive/refs/heads/master.zip

Title: Re: TextView
Post by: fearless on March 20, 2021, 11:47:55 PM
Another option could be to create a minimal font - basically just a font with the glyphs/symbols for windings etc that is required. Then embed the font into the control as data (hence the minimal size) and then use that font - as the font can be loaded as a resource once it is in memory.
Title: Re: TextView
Post by: Biterider on March 21, 2021, 06:40:56 PM
Hi
This is an interesting problem.
I started testing some other rendering APIs like TextOut and ExTextOut. They solved this problem with Wingdings 2, but introduced some new issues using fonts that behave fine with DrawText or DrawTextEx. In addition, TextOut requires kerning handling, which DrawText does automatically.

I've read some information and documentation on the internet, but it seems that there is no one-size-fits-all solution. :sad:

@fearless: your solution seems to be the most promising, but what puzzles me is that LiaoMi and I have the same font ("Wingdings 2", version 1.55), the same operating system and the same binary file and the result is very different.

I wonder if applications like MS-Word use their own rendering algorithms. I guess so...

Biterider
Title: Re: TextView
Post by: fearless on March 21, 2021, 09:29:49 PM
Couple of font making resources that are free (might require signup to website or mailing list but thats it)

Title: Re: TextView
Post by: LiaoMi on March 21, 2021, 09:58:32 PM
Quote from: Biterider on March 21, 2021, 06:40:56 PM
Hi
This is an interesting problem.
I started testing some other rendering APIs like TextOut and ExTextOut. They solved this problem with Wingdings 2, but introduced some new issues using fonts that behave fine with DrawText or DrawTextEx. In addition, TextOut requires kerning handling, which DrawText does automatically.

I've read some information and documentation on the internet, but it seems that there is no one-size-fits-all solution. :sad:

@fearless: your solution seems to be the most promising, but what puzzles me is that LiaoMi and I have the same font ("Wingdings 2", version 1.55), the same operating system and the same binary file and the result is very different.

I wonder if applications like MS-Word use their own rendering algorithms. I guess so...

Biterider

@Biterider

Apart from comparing systems, the last picture shows that the same text on different controls but in the same menu is displayed in different ways, how is this possible, very unusual, I did not notice this before  :biggrin:
Title: Re: TextView
Post by: Biterider on March 22, 2021, 12:13:58 AM
Hi
@fearless: thanks for the links. I checked them out. The most promising seems to be "fontforge".

@LiaoMi: I don't have a real explanation right now, but I'm still testing.
On my Win7 installation I get the same result as on Win10. On a raw Win8.1 installation, "Wingdings 2" is not installed by default. After installing it, I get the same result as on the other operating systems.
These are the machines I've access today. Tomorrow I will test on other PCs.

Biterider

Title: Re: TextView
Post by: fearless on March 22, 2021, 12:43:36 AM
Also found out about this Private Character Editor (eudcedit.exe) which comes with every version of windows: https://www.makeuseof.com/tag/create-your-own-fonts-symbols-characters-windows/
Title: Re: TextView
Post by: Biterider on March 22, 2021, 01:17:13 AM
Hi
I have never heard of this "Private Character Editor" before.
Incredible that it has always been there.  :biggrin:

Biterider
Title: Re: TextView
Post by: Biterider on March 22, 2021, 05:08:18 AM
Hi
Searching for more information I found a useful and free tool to visualize the Unicode mapping.
https://www.babelstone.co.uk/Software/BabelMap.html (https://www.babelstone.co.uk/Software/BabelMap.html)

Biterider
Title: Re: TextView
Post by: Biterider on May 14, 2021, 02:15:42 AM
Hi
I spent some time looking for a solution to the above issue.
Since nothing really worked and there was no useful information on the internet, I tried to find out for myself.

I use the DrawText API and the "Wingding 2" font to display some symbol characters in the TextView control. In particular, I was irritated to see missing glyphs near character 128 and above (empty square). My hope is that if I can solve this problem, I will have the solution for Liaomi's system too.

I started experimenting with the ExtTextOut API. It has a very useful flag called ETO_GLYPH_INDEX that I used to output all the characters in the font in sequence. To my surprise, all of the glyphs we can see in the "font insertion dialogbox" from a MS application were there, even when DrawText refuses to display them.
I suspected that DrawText was doing some sort of mapping when using some particular fonts.  :icon_idea:
I wrote a short program to show all characters wider than a predefined threshold and in the range 0000 to FFFFh and voilá. All the characters were there, but in very different positions!
I noticed that in the range from F000h to F0FFh all characters are in the correct order. That said, when we "and F000h" to the character value, we get the correct glyph from DrawText.

I don't have an explanation why this mapping is happening and I don't see any correlation with the Unicode BMP.  :sad:

This analysis is only valid for some sort of fonts. Other symbol fonts like Symbol, Webdings, Wingding, Wingding 3, behave the same way, while other regular fonts do not.

Changing the the charset to SYMBOL_CHARSET doesn't have any effect on the mapping.

If anyone has a clue on this topic, please post it here.

I changed the demo application to take this behavior into account. Additionally I show the characters in the range 0020h-00FFh and F020-F0FFh.
I was intrigued to see if the problem was gone on Liaomi's PC.

Biterider

Title: Re: TextView
Post by: jj2007 on May 14, 2021, 02:51:09 AM
Win7-64
Title: Re: TextView
Post by: LiaoMi on May 14, 2021, 06:30:19 AM
Quote from: Biterider on May 14, 2021, 02:15:42 AM
Hi
I spent some time looking for a solution to the above issue.
Since nothing really worked and there was no useful information on the internet, I tried to find out for myself.

I use the DrawText API and the "Wingding 2" font to display some symbol characters in the TextView control. In particular, I was irritated to see missing glyphs near character 128 and above (empty square). My hope is that if I can solve this problem, I will have the solution for Liaomi's system too.

I started experimenting with the ExtTextOut API. It has a very useful flag called ETO_GLYPH_INDEX that I used to output all the characters in the font in sequence. To my surprise, all of the glyphs we can see in the "font insertion dialogbox" from a MS application were there, even when DrawText refuses to display them.
I suspected that DrawText was doing some sort of mapping when using some particular fonts.  :icon_idea:
I wrote a short program to show all characters wider than a predefined threshold and in the range 0000 to FFFFh and voilá. All the characters were there, but in very different positions!
I noticed that in the range from F000h to F0FFh all characters are in the correct order. That said, when we "and F000h" to the character value, we get the correct glyph from DrawText.

I don't have an explanation why this mapping is happening and I don't see any correlation with the Unicode BMP.  :sad:

This analysis is only valid for some sort of fonts. Other symbol fonts like Symbol, Webdings, Wingding, Wingding 3, behave the same way, while other regular fonts do not.

Changing the the charset to SYMBOL_CHARSET doesn't have any effect on the mapping.

If anyone has a clue on this topic, please post it here.

I changed the demo application to take this behavior into account. Additionally I show the characters in the range 0020h-00FFh and F020-F0FFh.
I was intrigued to see if the problem was gone on Liaomi's PC.

Biterider

Hi Biterider,

very cool, great job! You can write an article, because even in windows there is a similar glitch with the display of squares. The picture shows that everything works fine. But I could not increase the height of the window  :biggrin:

(https://i.ibb.co/LPJ7gMx/2021-05-13-20-50-50-Text-View-Demo.jpg)

@jj2007 - did you manually change the height of the window?  :tongue:
P.S> Ah, now I see you glued two pictures together  :thup: Your test also has problems with displaying the font. But in the modification, all these problems have now been eliminated.
Title: Re: TextView
Post by: FORTRANS on May 14, 2021, 06:50:33 AM
Hi,

   Windows 10.  Cropped screenshot.

Steve
Title: Re: TextView
Post by: jj2007 on May 14, 2021, 07:03:29 AM
Quote from: LiaoMi on May 14, 2021, 06:30:19 AM@jj2007 - did you manually change the height of the window?  :tongue:
P.S> Ah, now I see you glued two pictures together  :thup: Your test also has problems with displaying the font. But in the modification, all these problems have now been eliminated.

PaintShop 4.1 is my friend :tongue:
Title: Re: TextView
Post by: nidud on May 14, 2021, 07:22:45 AM
deleted
Title: Re: TextView
Post by: Biterider on May 14, 2021, 03:18:46 PM
Hi
Thank you for testing. It seems to work fine, but it remains a mystery why Liaomi's system works differently.  :rolleyes:

@nidud: thanks, I'll check it.

Biterider
Title: Re: TextView
Post by: LiaoMi on May 14, 2021, 06:01:39 PM
Quote from: Biterider on May 14, 2021, 03:18:46 PM
but it remains a mystery why Liaomi's system works differently.  :rolleyes:

Hi Biterider,

Prologue>

Version 12.0.0 has been superseded by the latest version of the Unicode Standard - https://unicode.org/versions/Unicode12.0.0/
All Chapters and Appendices Together:
•   Full Text pdf for Viewing (14 MB) - http://www.unicode.org/versions/Unicode12.0.0/UnicodeStandard-12.0.pdf

Coded Character Sets, History and Development :
https://archive.org/details/mackenzie-coded-char-sets
https://archive.org/download/mackenzie-coded-char-sets/Mackenzie_CodedCharSets.pdf

Writing Unicode programs - GoAsm manual - http://www.godevtool.com/GoasmHelp/Unicode.htm

Main part

Why Non-Unicode apps system locale makes Unicode fonts with symbol charset displayed incorrectly? - https://stackoverflow.com/questions/21383099/why-non-unicode-apps-system-locale-makes-unicode-fonts-with-symbol-charset-displ

The root cause of the problem is that Wingdings font is actually non-Unicode font. It supports Unicode partially, so some symbols are still displayed correctly. See @Adrian McCarthy's answer for details about how it's probably works under the hood.

Also see more info here: http://www.fileformat.info/info/unicode/font/wingdings and here: http://www.alanwood.net/demos/wingdings.html

So what can we do to avoid such problems? I found several ways:

1. Quick & dirty

Fall back to ANSI version of API, as @user1793036 suggested:

TextOutA(hdc, 10, 10, "\x7d\x7e\x81\xfc"); // Displayed correctly!

2. Quick & clean

Use special Unicode range F0 (Private Use Area - https://en.wikipedia.org/wiki/Private_Use_Areas) instead of ASCII character codes. It's supported by Wingdings:

TextOutW(hdc, 10, 10, L"\xf07d\xf07e\xf081\xf0fc"); // Displayed correctly!
To explore which Unicode symbols are actually supported by font some font viewer can be used, e.g. dp4 Font Viewer

3. Slow & clean, but generic

But what to do if you don't know which characters you have to display and which font actually will be used? Here is most universal solution - draw text by glyphs to avoid any undesired translations:

void TextOutByGlyphs(HDC hdc, int x, int y, const CStringW& text)
{
   CStringW glyphs;

   GCP_RESULTSW gcpRes = {0};
   gcpRes.lStructSize = sizeof(GCP_RESULTS);
   gcpRes.lpGlyphs = glyphs.GetBuffer(text.GetLength());
   gcpRes.nGlyphs = text.GetLength();

   const DWORD flags = GetFontLanguageInfo(hdc) & FLI_MASK;
   GetCharacterPlacementW(hdc, text.GetString(), text.GetLength(), 0,
     &gcpRes, flags);
   glyphs.ReleaseBuffer(gcpRes.nGlyphs);

   ExtTextOutW(hdc, x, y, ETO_GLYPH_INDEX, NULL, glyphs.GetString(),
      glyphs.GetLength(), NULL);
}


TextOutByGlyphs(hdc, 10, 10, L"\x7d\x7e\x81\xfc"); // Displayed correctly!
Note GetCharacterPlacementW() function usage. For some unknown reason similar function GetGlyphIndicesW() would not work returning 'unsupported' dummy values for chars > 127.


@Adrian McCarthy's

Here's what I think is happening:

The Wingdings font doesn't have Unicode mappings (a cmap table?). (You can see this by using charmap.exe: the Character set drop down control is grayed out.)
For fonts without Unicode mappings, I think Windows assumes that it depends on the "Language for Non-Unicode applications" setting.
When that's English, Windows (probably) uses code page 1252, and all the values map to themselves.
When that's Russian, Windows (probably) uses code page 1251, and then tries to remap them.
The '\x81' value in code page 1251 maps to U+0403, which obviously doesn't exist in the font, so you get a box. Similarly the, '\xFC' maps to U+044C.
I assumed that if you used ExtTextOutW with the ETO_GLYPH_INDEX flag, Windows wouldn't try to interpret the values at all and just treat them as glyph indexes into the font. But that assumption is wrong.
However, there is another flag called ETO_IGNORELANGUAGE, which is reserved, but, empirically, it seems to solve the problem.

user1793036
I recall that on a Unicode build of my app some symbols would get the rectangle glyph, where in the Ansi build they would display fine. When I searched online I discovered that WingDings is only partially supported in Unicode. My solution was to draw each character using DrawTextA. This worked for me because I was only drawing individual characters.
Rost
Unfortunately ETO_IGNORELANGUAGE is not working, still same boxes are drawn.
Title: Re: TextView
Post by: Biterider on May 14, 2021, 10:57:06 PM
Hi LiaoMi
Thank you for pointing out this article. It has a lot of useful information!
According to the reasoning, the reason your system is responding differently may be due to the system's locale. So let's compare it. My is "Deutsch(Schweiz)"  :biggrin:

It seems that the reserved area (2. Quick & clean) is that what I found for these fonts.

Biterider
Title: Re: TextView
Post by: LiaoMi on May 15, 2021, 07:51:22 PM
Quote from: Biterider on May 14, 2021, 10:57:06 PM
Hi LiaoMi
Thank you for pointing out this article. It has a lot of useful information!
According to the reasoning, the reason your system is responding differently may be due to the system's locale. So let's compare it. My is "Deutsch(Schweiz)"  :biggrin:

It seems that the reserved area (2. Quick & clean) is that what I found for these fonts.

Biterider

Hi Biterider,

most likely it is true, https://i.ibb.co/vJBH7Qm/2021-05-15-10-48-45.jpg (https://i.ibb.co/vJBH7Qm/2021-05-15-10-48-45.jpg) and https://i.ibb.co/SVCRZyx/2021-05-15-11-44-08-Region.png (https://i.ibb.co/SVCRZyx/2021-05-15-11-44-08-Region.png) , https://i.ibb.co/5M1jJsX/2021-05-15-11-52-30-Text-View.jpg (https://i.ibb.co/5M1jJsX/2021-05-15-11-52-30-Text-View.jpg).

Yes, the second solution is like yours  :thumbsup:
Title: Re: TextView
Post by: Biterider on May 15, 2021, 09:13:59 PM
Hi LiaoMi
I tried your system configuration but could not reproduce your character mapping.
Using Windows in a language I don't know was a strange experience. I found most of the system interactions just looking at the icons  :tongue:

Biterider
Title: Re: TextView
Post by: LiaoMi on May 15, 2021, 09:32:33 PM
Quote from: Biterider on May 15, 2021, 09:13:59 PM
Hi LiaoMi
I tried your system configuration but could not reproduce your character mapping.
Using Windows in a language I don't know was a strange experience. I found most of the system interactions just looking at the icons  :tongue:

Biterider

:biggrin: I also did not succeed in repeating it with your configuration. Perhaps this does not work without a reboot, or these parameters are permanently saved to the registry (without reinstalling the system will not work?)
Title: Re: TextView
Post by: Biterider on May 15, 2021, 10:09:01 PM
I rebooted the system on all these last attempts and did not succeed.  :sad:
Title: Re: TextView
Post by: LiaoMi on May 15, 2021, 10:29:29 PM
Quote from: Biterider on May 15, 2021, 10:09:01 PM
I rebooted the system on all these last attempts and did not succeed.  :sad:

In this case, the problem is in the registry, this is the old code from Microsoft, which no one rewrote for new systems. If Unicode support is available on all systems, then you do not need to rewrite these modules, but for now it remains to live with it. 15 years ago there were the same problems, changing the encoding did not help, a rough change in the registry helped.

   Windows Registry Editor Version 5.00 
       
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage]   
    "1250"="c_1251.nls"   
    "1251"="c_1251.nls"   
    "1252"="c_1251.nls"   
    "1253"="c_1251.nls"   
    "1254"="c_1251.nls"   
    "1255"="c_1251.nls"     
     
    [HKEY_CURRENT_USER\Control Panel\International] 
    "Locale"="00000419" 
    "LocaleName"="ru-RU" 
    "s1159"="" 
    "s2359"="" 
    "sCountry"="Russia" 
    "sCurrency"="р." 
    "sDate"="." 
    "sDecimal"="," 
    "sGrouping"="3;0" 
    "sLanguage"="RUS" 
    "sList"=";" 
    "sLongDate"="d MMMM yyyy 'г.'" 
    "sMonDecimalSep"="," 
    "sMonGrouping"="3;0" 
    "sMonThousandSep"=" " 
    "sNativeDigits"="0123456789" 
    "sNegativeSign"="-" 
    "sPositiveSign"="" 
    "sShortDate"="dd.MM.yyyy" 
    "sThousand"=" " 
    "sTime"=":" 
    "sTimeFormat"="H:mm:ss" 
    "sShortTime"="H:mm" 
    "sYearMonth"="MMMM yyyy" 
    "iCalendarType"="1" 
    "iCountry"="7" 
    "iCurrDigits"="2" 
    "iCurrency"="1" 
    "iDate"="1" 
    "iDigits"="2" 
    "NumShape"="1" 
    "iFirstDayOfWeek"="0" 
    "iFirstWeekOfYear"="0" 
    "iLZero"="1" 
    "iMeasure"="0" 
    "iNegCurr"="5" 
    "iNegNumber"="1" 
    "iPaperSize"="9" 
    "iTime"="1" 
    "iTimePrefix"="0" 
    "iTLZero"="0" 
     
    [HKEY_CURRENT_USER\Control Panel\International\Geo] 
    "Nation"="203"


Some system files needed to be renamed. I don't think it's worth experimenting with your system these days. Someday this code will be rewritten  :biggrin:
Title: Re: TextView
Post by: fearless on September 02, 2021, 02:25:26 AM
Came across this font with glyphs - Segoe MDL2: https://docs.microsoft.com/en-us/windows/apps/design/style/segoe-ui-symbol-font

Only reference download I found is this: https://github.com/MicrosoftDocs/windows-uwp/issues/2320:


https://aka.ms/SegoeFonts - but appears to be an older version?



On Windows 10 it should be already installed, which you can check by viewing character map

It has glyphs for checkbox etc so could be useful. Also could include the font as a resource or as binary in a distribution and load font from memory/resource

Title: Re: TextView
Post by: Biterider on September 02, 2021, 03:54:31 AM
Hi fearless
Good catch. It is a very useful font, especially the fixed width allows the construction of many things, like the coloring described by MS in the link you provided.
It also has many advantages for the TextView control.  :cool:

Thanks for pointing it out.

Biterider
Title: Re: TextView
Post by: fearless on September 02, 2021, 06:00:40 AM
Yeh was looking at something similar for maybe use in some of the Modern UI stuff, maybe the caption bar and checkbox controls.
Also come across these fonts that have a lot of glyphs that could be useful as well:

https://github.com/Remix-Design/RemixIcon
https://github.com/icons8/line-awesome
Title: Re: TextView
Post by: fearless on September 04, 2021, 09:06:18 AM
Also came across this version that is not restricted license wise compared to Microsoft's Segoe UI and Segoe MDL2 assets - so is free to redistribute:

Has comparable assets to the Segoe MDL2 font and comes as a TTF. One version with popular web brands and the other without those brands.


http://xtoolkit.github.io/Micon/icons/

https://github.com/xtoolkit/Micon