The MASM Forum

General => The Campus => Topic started by: Mikl__ on August 11, 2017, 03:42:49 PM

Title: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: Mikl__ on August 11, 2017, 03:42:49 PM
Tutorial 35: RichEdit Control: Syntax Hilighting
Hi, all!
I collected an application from 35-th Iczelion lesson and found a strange and unpleasant mistake. If the comments are not Latin letters (in Russian Cyrillic is used), then everything looks like 00.png
In the attachment, the source text and exe-/rc-files from the site Iczelion https://win32assembly.programminghorizon.com/tut35.html
Tell me how can fix this error?
There is text with russian comments in tut_35.zip
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: aw27 on August 11, 2017, 04:41:45 PM
I don't know what nice and color rich editor you are using but I can see it very well with Notepad ++ and selecting one of the Cyrillic charsets.

StreamInProc proc hFile:QWORD,pBuffer:QWORD, NumBytes:QWORD, pBytesRead:QWORD
; Оo yЛoКВaМХЧ ЙoД ОpeДБopЪeРcЪ enter,40h,0 ОoЩРoЛy rbp=rsp+40h   
   and qword ptr [rbp-20h],0; mov qword ptr [rsp+20h],0
   invoke ReadFile; ,hFile,pBuffer,NumBytes,pBytesRead,0
   xor eax,1
   leave
   retn
StreamInProc endp
;--------------------------------
StreamOutProc proc hFile:QWORD,pBuffer:QWORD, NumBytes:QWORD, pBytesWritten:QWORD
; Оo yЛoКВaМХЧ ЙoД ОpeДБopЪeРcЪ enter,40h,0 ОoЩРoЛy rbp=rsp+40h   
   and qword ptr [rbp-20h],0; mov qword ptr [rsp+20h],0
   invoke WriteFile; ,hFile,pBuffer,NumBytes,pBytesWritten,0
   xor eax,1
   leave
   retn
StreamOutProc endp
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: jj2007 on August 11, 2017, 06:30:58 PM
Here is a little converter from Russian (codepage 1251) to UTF8. Just drag your source over the attached exe.

include \masm32\MasmBasic\MasmBasic.inc
  Init
  Let esi=Utf8$(wCL$())  ; Unicode file names are allowed
  Let edi=Left$(esi, Instr_(esi, ".")-1)+"converted.asm"
  Open "O", 1, edi
  Print #1, Utf8Bom
  Recall esi, L$()
  For_ ecx=0 To eax-1
        PrintLine #1, ConvertCp$(L$(ecx), 1251, CP_UTF8)
  Next
  Close
  ShEx edi
EndOfCode

Code: [Select]
; пo yмoлчaнию кoд пpeдвopяeтcя enter,40h,0 пoэтoмy rbp=rsp+40h
; зaпoлняeм пoля cтpyктypы CHARFORMAT, чтoбы ycтaнoвить цвeт тeкcтa
...
;=============================================================
;  Кoгдa пoльзoвaтeль кликaeт нa oднoй из пaлитpы цвeтoв, зaпoлняeм пoля
;  cтpyктypы CHOOSECOLOR и вызывaeм диaлoгoвoe oкнo выбopa цвeтa ChooseColor.
;  Еcли пoльзoвaтeль выбиpaeт цвeт, тo этo знaчeниe colorref вoзвpaщaeтcя в
;  члeнe rgbResult, coхpaняeм этo знaчeниe в пepeмeннoй BackgroundColor.
;  Пocлe этoгo, мы вынyждaeм пepeкpaшивaниe нa пaлитpe цвeтoв, вызывaя
;  InvalidateRect нa дecкpиптop пaлитpы цвeтoв. Пaлитpa цвeтoв пocылaeт
;  WM_CTLCOLORSTATIC cooбщeниe poдитeльcкoмy oкнy
...
;==========================================
;  Кoгдa пoльзoвaтeль кликaeт нa oднoй из пaлитpы цвeтoв, вызывaeтcя диaлoгoвoe
;  oкнo выбopa цвeтa. "Пaлитpa цвeтoв" - cтaтичecкий элeмeнт yпpaвлeния c флaгoм
;  WS_BORDER и SS_NOTIFY. Стaтичecкий элeмeнт yпpaвлeния c флaгoм SS_NOTIFY
;  yвeдoмит poдитeльcкoe oкнo o дeйcтвиях мыши нa нeм, типa BN_CLICKED
;  (STN_CLICKED). Этo - yлoвкa.
;===========================================
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: hutch-- on August 11, 2017, 10:34:08 PM
I remember doing a syntax colouring version back in about 2000 using Iczelion's technique and it worked well as it was only producing technicolor for the client area and not modifying the entire file. This made it fast and the normal capacity with richedit 2/3 to load very large files (>100 meg) FAST !!!!

I never followed it up because I just don't like technicolor in source code, I find it very distracting and much slower to read.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: HSE on August 11, 2017, 11:56:23 PM
One or two years ago I tried to use this Iczelion's rutines in a plugin for qEditor. Of course not sucess at all :(
The almost sure thing is that I maked everything wrong  :biggrin:, but look like qEditor set colors after the entry point and a plugins can't to change colors. 
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: hutch-- on August 12, 2017, 12:33:19 AM
You are right, it will not work in QE as it has its own colour control. If you wanted to try Iczelion's technique I would try it in a basic rich edit control that does not have its own colour control and it should work OK. Its been a long time ago but it was a fast technique and worked well.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: HSE on August 12, 2017, 01:32:25 AM
Thanks Hutch! It's encouraging to know that the failure is not  totally my fault  :biggrin:

The Iczelion's example it's a complete editor, only very simple. Would be nice Iczelion's colouring in qEditor, just take in count if someday you rework qEditor. For years I used Notepad++. At that time there was no IDE availables that worth the price, and Notepad++ was in the middle between a simple editor and a IDE. For complex projects I'm using RadAsm (and there are others very good IDE for that). For some time now, I prefer qEditor options for single files, and it's the default application for *.asm, *.inc, *.txt, *.ini and some others.  :t   

Later: Notepad++
   
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: mineiro on August 12, 2017, 09:40:20 AM
That tutorial techniquè acts like a transparent frame in front of text, you can reproduce that effect if you change text size by control+mouse whell, the font being used need be sync with text+frame.

I have tried gtk+ and sounds good, easy, well documented, they separated text buffer and text viewer. The thing is based on markup language, supports utf8 and many string conversion routines. I have also an idea about that on decomposed unicode chars, I don't have seen text editors saving unicode chars on decomposed way. I think this can be a solution not only to utf8/16.... encode way but others that can born on future.
Imagine that you're reading a text and you markup all foreginners words on that text, like per example latim words being italics. This way, we can search words on that text not by finding words but styles, and you can remove or list all latim words from that text. Other idea was about accents, but I lost my libid on that moment when I feel that this need be coded. We cannot search for accents only, we actually need search for 'áéíóú' instead of only accents. Well, If I'm a cyrillic writer today I probably adopt utf8 as jj2007 told, or utf16 like aw27. I searched and have not only utf encode, on Asia they are using another encode way.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: hutch-- on August 12, 2017, 11:18:54 AM
The problem is with editors for code is that code has no reason to be written in unicode which generates the problem for folks whose native language cannot be represented using a 256 characters set. Now you can go the way of writing the source in unicode and converting it to ascii to compile or assemble it or you can vertically split the display so that the code is in ascii and the side comments are in unicode. This would be a synchronised pair of edit windows with a splitter bar down the centre.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: nidud on August 12, 2017, 01:12:38 PM
The problem is with editors for code is that code has no reason to be written in unicode which generates the problem for folks whose native language cannot be represented using a 256 characters set.

There's only a selected few languages that can't be represented using a 256 characters set. Russian is not one of them so it must be something else.

Quote
Now you can go the way of writing the source in unicode and converting it to ascii to compile or assemble it or you can vertically split the display so that the code is in ascii and the side comments are in unicode. This would be a synchronised pair of edit windows with a splitter bar down the centre.

The source code is written in ASCII so I think it safe to assume it looks and works correctly in qeditor, notepad and other IDE's.

I noticed from using the editor that some highlighted text has a different size than the background but judging from the image attached there seems to be a font problem.

Quote
The rich edit control uses Unicode internally, so this use of character sets differs from the original one used in font specifications. But the CHARFORMAT (https://msdn.microsoft.com/en-us/library/windows/desktop/bb787881(v=vs.85).aspx) structure has a well-defined place for the character set.

However, there must be some different handling for the editor than for the highlighted text given the background is correctly displayed (I assume), so changing the font may not fix it.

You may try to manually convert the strings and see if that helps.

Code: [Select]
if 1
invoke lstrlen,edi
lea edx,wbuffer
mov word ptr [edx+eax*2],0
invoke MultiByteToWideChar,866,0,edi,eax,edx,eax
invoke DrawTextW,hdc,addr wbuffer,-1,addr rect,0
else
invoke DrawText,hdc,edi,-1,addr rect,0
endif
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: nidud on August 12, 2017, 01:46:14 PM
I noticed from using the editor that some highlighted text has a different size than the background but judging from the image attached there seems to be a font problem.
...
However, there must be some different handling for the editor than for the highlighted text given the background is correctly displayed (I assume), so changing the font may not fix it.

So, the font used by the highlighted text is not the same as the background. The original font must then have been created by using CreateFont() or a similar function returning a handle.

Code: [Select]
NewRichEditProc proc hWnd:DWORD, uMsg:DWORD, wParam:DWORD, lParam:DWORD
LOCAL hdc:DWORD
LOCAL hOldFont:DWORD

Code: [Select]
invoke SelectObject,hdc,hOldRgn
invoke DeleteObject,hRgn
invoke SelectObject,hdc,hOldFont

The SelectObject function selects an object into the specified device context (DC). The new object replaces the previous object of the same type. hOldFont is not initialized so maybe that's the problem?
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: aw27 on August 12, 2017, 02:03:53 PM
Well, If I'm a cyrillic writer today I probably adopt utf8 as jj2007 told, or utf16 like aw27.
Actually, I never proposed to save source code as UTF16 and I am firmly against that.  :icon_eek:
Probably what I said was that the assembler must be able to generate UTF16  at assembly time (I am not talking about adding a 0 to an ASCII character) and it can only do that if the source code is saved as UTF8 (or other Unicode form, like UTF16, of course) when it contains characters not available in the local encoding schema.

This is actually what  compilers like Visual Studio do. When you start writing a program in VS it assumes your local character encoding, be it Latin-1, Greek or Shift JIS.
As soon as you type a character not available in the local encoding schema and save the source, VS will alert you (with a message box) that the source must be saved in UTF8 to preserve the integrity.


Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: aw27 on August 12, 2017, 02:19:17 PM
Notepad was in the middle between a simple editor and a IDE.
Notepad is much lower than middle, you can not descend more.
Don't confuse Notepad with Notepad++, which is a true powerful "simple editor". Note the contradiction between  powerful and  simple.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: minor28 on August 12, 2017, 04:53:31 PM
A few years ago I wrote a project based on richedit control (msftedit.dll).

Source code at https://minor28.divdev.se/
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: jj2007 on August 12, 2017, 06:25:07 PM
A few years ago I wrote a project based on richedit control (msftedit.dll).

Looks nice but I encountered a problem: When I saved an asm source as rtf, and tried to reopen it, I had a blank page in front of me. Toolbar, menus etc all ok but no document. Then, after a while, the message below popped up, but at a moment when the editor was no longer an active process. Now, when I keep trying, the editor opens fine but doesn't open any documents.

This is Win7-64. Weird ::)
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: nidud on August 13, 2017, 01:41:42 AM
Here's a test case using the font face from the Rich Edit handle. It's a modified version of the windows/3/test.asm (https://github.com/nidud/asmc/blob/master/source/test/windows/3/test.asm) file using Rich Edit version 4.1.

Code: [Select]
;
; https://msdn.microsoft.com/en-us/library/windows/desktop/dd183499(v=vs.85).aspx
; https://msdn.microsoft.com/en-us/library/windows/desktop/hh298375(v=vs.85).aspx
;
include windows.inc

ifdef _UNICODE
MSFTEDIT_CLASS equ <L"RICHEDIT50W">
else
MSFTEDIT_CLASS equ <"RICHEDIT50W">
endif

.data
hEdit HWND ?

.code

CreateRichEdit proc hwndOwner:HWND, ; Dialog box handle.
                    x:SINT, y:SINT, ; Location.
                    w:SINT, h:SINT, ; Dimensions.
                    hinst:HINSTANCE ; Application or DLL instance.

    LoadLibrary("Msftedit.dll")
    CreateWindowEx(0, MSFTEDIT_CLASS,
        "Type here\n"
        "ñîçäàäèì äî÷åðíåå îêíî ñëàéäåðà 2",
        ES_MULTILINE or WS_VISIBLE or WS_CHILD or WS_BORDER or WS_TABSTOP,
        x, y, w, h,
        hwndOwner, NULL, hinst, NULL)
    ret

CreateRichEdit endp

WndProc proc hWnd:HWND, message:UINT, wParam:WPARAM, lParam:LPARAM

    local wmId, wmEvent
    local ps:PAINTSTRUCT
    local hdc:HDC
    local rect:RECT
    local hBrush:HBRUSH
    local hFont:HFONT

    .switch message

      .case WM_PAINT

        mov hdc,BeginPaint(hWnd, &ps)

        ; Get the font face from Rich Edit handle.

        mov hFont,SendMessage(hEdit, WM_GETFONT, 0, 0)
        SelectObject(hdc, hFont)

        ; Sets the coordinates for the rectangle in which the text is to be formatted.
        SetRect(&rect, 10,70,500,50)
        SetTextColor(hdc, RGB(255,0,0))
        DrawText(hdc, "ñîçäàäèì äî÷åðíåå îêíî ñëàéäåðà 2", -1, &rect, DT_NOCLIP)


        ; Logical units are device dependent pixels, so this will create a
        ; handle to a logical font that is 36 pixels in height.
        ; The width, when set to 20, will cause the font mapper to choose a font
        ; which, in this case, is stretched.
        ; The font face name will be Times New Roman.  This time nEscapement is
        ; at -300 tenths of a degree (-30 degrees)
        mov hFont, CreateFont(36,20,-300,0,FW_DONTCARE,FALSE,TRUE,FALSE,DEFAULT_CHARSET,OUT_OUTLINE_PRECIS,
                CLIP_DEFAULT_PRECIS,CLEARTYPE_QUALITY, VARIABLE_PITCH, "Times New Roman")
        SelectObject(hdc, hFont)

        ; Sets the coordinates for the rectangle in which the text is to be formatted.
        SetRect(&rect, 100, 200, 900, 800);
        SetTextColor(hdc, RGB(0,128,0));
        DrawText(hdc, "ñîçäàäèì äî÷åðíåå îêíî ñëàéäåðà 2", -1, &rect, DT_NOCLIP)

        ; Logical units are device dependent pixels, so this will create a handle
        ; to a logical font that is 36 pixels in height.
        ; The width, when set to 10, will cause the font mapper to choose a font
        ; which, in this case, is compressed.
        ; The font face name will be Arial. This time nEscapement is at 250
        ; tenths of a degree (25 degrees)

        mov hFont,CreateFont(36,10,250,0,FW_DONTCARE,FALSE,TRUE,FALSE,DEFAULT_CHARSET,
            OUT_OUTLINE_PRECIS,CLIP_DEFAULT_PRECIS,ANTIALIASED_QUALITY,VARIABLE_PITCH,
            "Arial")
        SelectObject(hdc,hFont)

        ; Sets the coordinates for the rectangle in which the text is to be formatted.
        SetRect(&rect, 500, 200, 1400, 600)
        SetTextColor(hdc, RGB(0,0,255))
        DrawText(hdc, "ñîçäàäèì äî÷åðíåå îêíî ñëàéäåðà 2", -1, &rect, DT_NOCLIP)
        DeleteObject(hFont)

        EndPaint(hWnd, &ps)
        xor eax,eax
        .endc

      .case WM_DESTROY
        PostQuitMessage(0)
        xor eax,eax
        .endc
      .default
        DefWindowProc(hWnd, message, wParam, lParam)
    .endsw
    ret
WndProc endp

WinMain proc WINAPI hInstance: HINSTANCE,
     hPrevInstance: HINSTANCE,
         lpCmdLine: LPSTR,
          nShowCmd: SINT

    local wc:WNDCLASSEX
    local msg:MSG
    local hwnd:HANDLE

    mov wc.cbSize,SIZEOF WNDCLASSEX
    mov wc.style,CS_HREDRAW or CS_VREDRAW
    mov wc.lpfnWndProc,WndProc

    mov ecx,hInstance
    xor eax,eax
    mov wc.cbClsExtra,eax
    mov wc.cbWndExtra,eax
    mov wc.hInstance,ecx
    mov wc.hbrBackground,COLOR_WINDOW+1
    mov wc.lpszMenuName,eax

    lea eax,@CStr("WndClass")
    mov wc.lpszClassName,eax
    mov wc.hIcon,LoadIcon(0, IDI_APPLICATION)
    mov wc.hIconSm,eax
    mov wc.hCursor,LoadCursor(0, IDC_ARROW)

    RegisterClassEx(&wc)

    mov eax,CW_USEDEFAULT
    mov hwnd,CreateWindowEx(0, "WndClass", "Window", WS_OVERLAPPEDWINDOW,
        eax, eax, eax, eax, 0, 0, hInstance, 0)

    mov hEdit,CreateRichEdit(eax, 10, 10, 500, 50, hInstance)

    ShowWindow(hwnd, SW_SHOWNORMAL)
    UpdateWindow(hwnd)

    .while GetMessage(&msg, 0, 0, 0)
        TranslateMessage(&msg)
        DispatchMessage(&msg)
    .endw
    mov eax,msg.wParam
    ret

WinMain endp

ifdef __PE__
WinStart proc
    mov ebx,GetModuleHandle(0)
    ExitProcess(WinMain(ebx, 0, GetCommandLineA(), SW_SHOWDEFAULT))
WinStart endp
else
WinStart equ <>
endif
    end WinStart

So if you could run the exe-file Mikl__ and post the image of the window output we will at least know if it's a font problem.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: HSE on August 13, 2017, 02:01:10 AM
I noticed from using the editor that some highlighted text has a different size than the background but judging from the image attached there seems to be a font problem.
There is no problem in the original 32bit.
 
Don't confuse Notepad with Notepad++...
:biggrin: :biggrin: I totally forget "Bloc de notas", corrected now.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: nidud on August 13, 2017, 02:27:16 AM
I noticed from using the editor that some highlighted text has a different size than the background..
There is no problem in the original 32bit.

Maybe it depends on the local font?
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: HSE on August 13, 2017, 02:53:20 AM
Maybe it depends on the local font?

It's very easy to answer that: I don't know.  :eusa_snooty:
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: nidud on August 13, 2017, 04:07:51 AM
It's very easy to answer that: I don't know.  :eusa_snooty:

 :biggrin:

You could sort-of just write ;Copy & style in the editor before you took the snapshot thought. If you really wanted to know the answer that is.
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: HSE on August 13, 2017, 04:33:47 AM
The ampersand make the crash jump  :t
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: nidud on August 17, 2017, 09:34:52 AM
The ampersand make the crash jump  :t

DrawText(hdc, txtrange.lpstrText, -1, &rect, DT_NOPREFIX)

https://msdn.microsoft.com/en-us/library/windows/desktop/dd162498(v=vs.85).aspx

Quote
Example:
input string: "A&bc&&d"
normal: "Abc&d"
DT_NOPREFIX: "A&bc&&d"
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: HSE on August 17, 2017, 11:39:48 PM
Impressive  :t

(Obviously not problem with EM_GETTEXTRANGE  :biggrin: :biggrin:)
Title: Re: Iczelion Tutorial 35: RichEdit Control: Syntax Hilighting
Post by: Mikl__ on August 24, 2017, 11:51:12 PM
Thanks to all for help. I solved the problem, if anybody it's interested then see in Uncle Remus Tales (http://masm32.com/board/index.php?topic=6275.msg69642#msg69642)