The MASM Forum

Hi to all! How convert UTF-16 string to UTF-8 or OEMCP code (FindNextFileW)?

Quote from: bomz on February 25, 2025, 05:52:29 AMHi to all! How convert UTF-16 string to UTF-8 or OEMCP code (FindNextFileW)?

Hi bomz. I had asked that your post be made into its own topic, where you might get a better response than posting your question to an unrelated existing topic.

I myself do not have an answer for you, as I work solely with ascii, 99.9 percent of the time.

[deleted]

problem occurred when downloading video files from YouTube, windows can't reflect files name, but can address files - delete,move, copy, rename, back rename. where is no method how do it in console, only use mass method with musk *.mp4

For UTF-8, you could try WideCharToMultiByte (https://learn.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-widechartomultibyte).

Quote from: bomz on February 25, 2025, 11:21:20 AMproblem occurred when downloading video files from YouTube, windows can't reflect files namess method with musk *.mp4

Did this just start recently, or has it always been this way? I am just curious.

[deleted]

Quote from: NoCforMe on February 25, 2025, 10:57:49 AMIf the 16-bit text is anything like Unicode, which I suspect it is, you can simply do the "read a word, write a byte" method:

Code Select Expand

; Assuming ECX points to your UTF-16 string,
; and EDX points to where you want to store the ASCII:

next: MOV AX, [ECX] ;Read a WORD.
MOV [EDX], AL ;Write a BYTE.
TEST AX, AX ;If it's a zero,
JZ done ;  that's the end.
ADD ECX, 2 ;Advance your pointers.
INC EDX
JMP next

done:

You don't have to use those registers; I just wrote the code that one way.
But you get the idea.

That is so wrong it's giving me double vision :dazzled:
Try it on real unicode, like some chinese text.

Well, if that won't work, you'll need to use some kind of translation table to convert UTF-16, right?

Quote from: sinsi on February 25, 2025, 12:03:29 PM

Quote from: NoCforMe on February 25, 2025, 10:57:49 AMIf the 16-bit text is anything like Unicode, which I suspect it is, you can simply do the "read a word, write a byte" method:

Code Select Expand

; Assuming ECX points to your UTF-16 string,
; and EDX points to where you want to store the ASCII:

next: MOV AX, [ECX] ;Read a WORD.
MOV [EDX], AL ;Write a BYTE.
TEST AX, AX ;If it's a zero,
JZ done ;  that's the end.
ADD ECX, 2 ;Advance your pointers.
INC EDX
JMP next

done:

You don't have to use those registers; I just wrote the code that one way.
But you get the idea.

That is so wrong it's giving me double vision :dazzled:
Try it on real unicode, like some chinese text.

Looks like unicode->ascii conversion code.

QuoteZài zhēnzhèng de unicode shàng chángshì yīxià, lìrú yīxiē zhōngwén wénzì.

5A C3 A0 69 20 7A 68 C4 93 6E 7A 68 C3 A8 6E 67 20 64 65 20 75 6E 69 63 6F 64 65 20 73 68 C3 A0
6E 67 20 63 68 C3 A1 6E 67 73 68 C3 AC 20 79 C4 AB 78 69 C3 A0 2C 20 6C C3 AC 72 C3 BA 20 79 C4
AB 78 69 C4 93 20 7A 68 C5 8D 6E 67 77 C3 A9 6E 20 77 C3 A9 6E 7A C3 AC 2ETake every second byte away and you're still left with gibberish :biggrin:

Quote from: NoCforMe on February 25, 2025, 12:26:28 PMWell, if that won't work, you'll need to use some kind of translation table to convert UTF-16, right?

Quote from: tenkey on February 25, 2025, 11:24:53 AMFor UTF-8, you could try WideCharToMultiByte (https://learn.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-widechartomultibyte).

Even WideCharToMultiByte isn't guaranteed to convert properly, or convert to an ANSI string (as the ToMultiByte part infers).

So how does that stuff get translated to ASCII? or can it even be?

Converting Between Unicode UTF-16 and UTF-8 Using ATL CString and Direct Win32 API Calls (https://giodicanio.com/2023/09/27/converting-between-unicode-utf-16-and-utf-8-using-atl-cstring-and-direct-win32-api-calls/)
I try repeat, but something wrong. In the same time I have UTF-8 to OEMCP working code
(https://i.postimg.cc/ZCyPcvYD/sshot-3.jpg) (https://postimg.cc/ZCyPcvYD)

Quote from: sinsi on February 25, 2025, 12:03:29 PMThat is so wrong it's giving me double vision :dazzled:
Try it on real unicode, like some chinese text.

Indeed. All file names are stored internally as Utf-16. You either use them "as is" in Utf-16, or you convert them for display purposes to Utf-8.

Quote from: tenkey on February 25, 2025, 11:24:53 AMFor UTF-8, you could try WideCharToMultiByte (https://learn.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-widechartomultibyte).

Exactly.

perhaps I am wrong to use console streams?

Quote from: bomz on February 25, 2025, 11:56:40 PMperhaps I am wrong to use console streams?

Usually no problem, bomz, but consoles tend to display only locally common character sets. In Europe you may be able to display cyrillic text but no chinese, in China the console will surely display chinese and english text but probably no Russian.

If you want to see all kinds of characters, use a static, edit or richedit control.

problem is not to see text, which can't be reflect in console, directory list redirect to file, but save true name to file after download and ffmpeg = have possibility to rename and back rename, copy, move, delete....

Quote from: bomz on February 26, 2025, 08:16:24 AMhave possibility to rename and back rename, copy, move, delete....

No problem if you use systematically the xxxW functions and Utf16 :thumbsup:

    invoke MultiByteToWideChar,CP_UTF8,0,hMemPointer,ebx,0,0
    mov _bs, eax
    invoke MultiByteToWideChar,CP_UTF8,0,hMemPointer,ebx,Buffer,esi
    invoke WideCharToMultiByte,CP_OEMCP,0,Buffer,_bs,hMemPointer,SizeofhMemory,0,0if try to use as suggests above MultiByteToWideChar, code for convert UTF-8 to OEMCP must work with UTF-16 too. If try to change CP_UTF8 to 1200 error occurs, because no UTF-16 codepage in Windows, what is not surprise because Windows can't reflect such names correctly. Look like Windows api can't work with UTF-16 only transfer UTF-16 names to nuclear function. If try transfer UTF-16 code from one Windows api to another or try use console streams code destroys. or something like that

Quote from: bomz on February 28, 2025, 01:22:18 AMMultiByteToWideChar... must work with UTF-16 too.

Well, not really: if your source is CP_UTF16, it is already wide char... nothing to convert :cool:

First I do not try convert anything, just direct console stream to file, for back rename after ffmpeg. as result I want get pull of console utilities which allow work with such files

Quote from: bomz on February 28, 2025, 03:51:38 AMFirst I do not try convert anything, 

But the title of this thread clearly says something different. "How convert UTF-16 string to UTF-8 or OEMCP code"

Do you see how this can cause confusion?

exactly. try do it

Quote from: bomz on February 28, 2025, 01:22:18 AMMultiByteToWideChar, code for convert UTF-8 to OEMCP must work with UTF-16

Quote from: bomz on February 28, 2025, 03:51:38 AMI do not try convert anything

Please take a decision :cool:

Q: How convert UTF-16 string to UTF-8 and to OEMCP code , direct console stream to file, use MASM64 assembly for windows 10

A: To achieve the tasks of converting a UTF-16 string to UTF-8, converting it further to OEM code page (OEMCP), and redirecting the console stream to a file using MASM64 assembly on Windows 10, we need to break the problem into several steps. Each step involves interacting with the Windows API, as MASM64 does not provide built-in functions for these operations.
Below is a detailed explanation of the process, followed by an example implementation in MASM64 assembly:

Step-by-Step Explanation

1. Convert UTF-16 String to UTF-8
The Windows API provides the WideCharToMultiByte function, which can be used to convert a UTF-16 (wide character) string to a UTF-8 encoded string. The function requires specifying the code page as CP_UTF8.

2. Convert UTF-8 String to OEM Code Page
Once the string is in UTF-8 format, you can use the same WideCharToMultiByte function again, but this time specify the OEM code page (e.g., CP_OEMCP) to convert the UTF-8 string to the OEM code page.

3. Redirect Console Stream to a File
To redirect the console output to a file, you can use the CreateFile function to open or create a file, and then use SetStdHandle to redirect the standard output (STD_OUTPUT_HANDLE) to the file handle.

4. Write to the Console/File
Finally, you can use the WriteConsole or WriteFile function to write the converted string to the redirected console/file.

MASM64 Assembly Implementation
Below is an example implementation in MASM64 assembly that performs the above steps:
; MASM64 Example: Convert UTF-16 to UTF-8, then to OEMCP, and redirect console output to a file
include \masm64\include\masm64rt.inc
.data
    utf16_string db "Hello, World!", 0          ; UTF-16 string (null-terminated)
    utf8_buffer db 256 dup(0)                   ; Buffer for UTF-8 conversion
    oem_buffer db 256 dup(0)                    ; Buffer for OEMCP conversion
    file_handle HANDLE ?                        ; Handle for the output file
    bytes_written DWORD ?                       ; Number of bytes written
.code
main proc
 ; Step 1: Convert UTF-16 to UTF-8
 invoke WideCharToMultiByte, CP_UTF8, 0, offset utf16_string, -1, offset utf8_buffer, sizeof utf8_buffer, NULL, NULL
 ; Step 2: Convert UTF-8 to OEMCP
 invoke MultiByteToWideChar, CP_UTF8, 0, offset utf8_buffer, -1, offset oem_buffer, sizeof oem_buffer
 invoke WideCharToMultiByte, CP_OEMCP, 0, offset oem_buffer, -1, offset oem_buffer, sizeof oem_buffer, NULL, NULL
 ; Step 3: Open/Create a file for output
 invoke CreateFile, "output.txt", GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL
mov file_handle, rax
; Step 4: Redirect console output to the file
invoke SetStdHandle, STD_OUTPUT_HANDLE, file_handle
; Step 5: Write the OEMCP string to the file
invoke GetStdHandle, STD_OUTPUT_HANDLE
invoke WriteFile, rax, offset oem_buffer, sizeof oem_buffer, addr bytes_written, NULL
 ; Cleanup
 invoke CloseHandle, file_handle
 ; Exit program
 invoke ExitProcess, 0
main endp
end ; MASM64 Example: Convert UTF-16 to UTF-8, then to OEMCP, and redirect console output to a file

Explanation of Key Functions
WideCharToMultiByte :
Converts a wide character string (UTF-16) to a multibyte string (UTF-8 or OEMCP).
Parameters:
CodePage: Specifies the target code page (e.g., CP_UTF8 or CP_OEMCP).
dwFlags: Conversion flags (usually 0).
lpWideCharStr: Pointer to the source UTF-16 string.
cchWideChar: Length of the source string (-1 for null-terminated).
lpMultiByteStr: Pointer to the buffer for the converted string.
cbMultiByte: Size of the buffer.
lpDefaultChar: Default character for unmappable characters (optional).
lpUsedDefaultChar: Indicates if default characters were used (optional).
MultiByteToWideChar :
Converts a multibyte string (UTF-8) back to a wide character string (UTF-16).
Used here to prepare the UTF-8 string for conversion to OEMCP.
CreateFile :
Creates or opens a file for writing.
Returns a handle to the file.
SetStdHandle :
Redirects the standard output handle to the specified file handle.
WriteFile :
Writes data to the file or redirected console.
CloseHandle :
Closes the file handle after writing.

Quote from: ognil on February 28, 2025, 05:33:48 AM; Step 1: Convert UTF-16 to UTF-8
 invoke WideCharToMultiByte, CP_UTF8, 0, offset utf16_string, -1, offset utf8_buffer, sizeof utf8_buffer, NULL, NULL
 ; Step 2: Convert UTF-8 to OEMCP
 invoke MultiByteToWideChar, CP_UTF8, 0, offset utf8_buffer, -1, offset oem_buffer, sizeof oem_buffer

So, in step 1 you convert Utf-16 to Utf-8. Then, in step 2, you convert the converted string to CP_OEM (https://learn.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-widechartomultibyte). That's genius :thumbsup:

(minor criticism: the "oem_buffer" should be named "wide_buffer" because that's what it gets)

it is impossible do with using console streams. Only through write direct to file to save UTF-16 code

Using your executable. Where is the source btw?

*

*

Tmp_File.asm(8) : Error A2106: Cannot open file: "\masm32\macros\Strings.mac" [ENOENT]
Tmp_File.asm(9) : Error A2106: Cannot open file: "\MASM32\INCLUDE\w2k\ntddkbd.inc" [ENOENT]
Tmp_File.asm(10) : Error A2106: Cannot open file: "\MASM32\INCLUDE\w2k\w2kundoc.inc" [ENOENT]
Tmp_File.asm(11) : Error A2106: Cannot open file: "\MASM32\INCLUDE\winioctl.inc" [ENOENT]
Tmp_File.asm(26) : Error A2210: Syntax error: ClientId
Tmp_File.asm(27) : Error A2210: Syntax error: ImageInformation
Tmp_File.asm(90) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(91) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(92) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(93) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(94) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(95) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(96) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(97) : Error A2210: Syntax error: CCOUNTED_UNICODE_STRING
Tmp_File.asm(98) : Error A2210: Syntax error: CTW0
Tmp_File.asm(99) : Error A2210: Syntax error: CTW0
Tmp_File.asm(100) : Error A2210: Syntax error: CTW0
Tmp_File.asm(107) : Error A2210: Syntax error: usString
Tmp_File.asm(108) : Error A2210: Syntax error: usBuffer
Tmp_File.asm(109) : Error A2210: Syntax error: usNextChar
Tmp_File.asm(112) : Error A2210: Syntax error: TW
Tmp_File.asm(113) : Error A2210: Syntax error: TW
Tmp_File.asm(121) : Error A2210: Syntax error: TW0
Tmp_File.asm(125) : Error A2210: Syntax error: TW0
Tmp_File.asm(129) : Error A2210: Syntax error: TW0
Tmp_File.asm(141) : Error A2210: Syntax error: usstring
Tmp_File.asm(142) : Error A2210: Syntax error: KeyBoardAttrib
Tmp_File.asm(144) : Error A2210: Syntax error: Iosb
Tmp_File.asm(148) : Error A2210: Syntax error: KeyBoardBuffer
Tmp_File.asm(150) : Error A2210: Syntax error: KeyBoardIndicator
Tmp_File.asm(153) : Error A2210: Syntax error: DirectoryAttrib
Tmp_File.asm(157) : Error A2210: Syntax error: usnt_file
Tmp_File.asm(158) : Error A2210: Syntax error: usimgname
Tmp_File.asm(159) : Error A2210: Syntax error: usdllpath
Tmp_File.asm(160) : Error A2210: Syntax error: uscmdline
Tmp_File.asm(161) : Error A2210: Syntax error: usimgpath
Tmp_File.asm(162) : Error A2210: Syntax error: processparameters
Tmp_File.asm(177) : Error A2210: Syntax error: FileAttrib1
Tmp_File.asm(179) : Error A2210: Syntax error: usFile1
Tmp_File.asm(180) : Error A2210: Syntax error: FileAttrib2
Tmp_File.asm(182) : Error A2210: Syntax error: usFile2
Tmp_File.asm(187) : Error A2210: Syntax error: VolumeAttrib
Tmp_File.asm(191) : Error A2210: Syntax error: RegistryAttrib
Tmp_File.asm(195) : Error A2210: Syntax error: NativeDriverAttrib
Tmp_File.asm(198) : Error A2210: Syntax error: DiskAttrib
Tmp_File.asm(202) : Error A2210: Syntax error: time2
Tmp_File.asm(218) : Error A2102: Symbol not defined : PPEB
Tmp_File.asm(226) : Error A2160: INVOKE requires prototype for procedure
Tmp_File.asm(233) : Error A2160: INVOKE requires prototype for procedure
Tmp_File.asm(243) : Error A2210: Syntax error: InitializeObjectAttributes
Tmp_File.asm(256) : Error A2210: Syntax error: InitializeObjectAttributes
Tmp_File.asm(256) : Fatal error A1113: Too many errors
*** Assembly error ***

look it is need lstutf-16tofile.exe and renamefromfileutf-16.exe
*

Quote from: bomz on March 09, 2025, 12:01:31 AM*

*

??? What is all of that for?
Are you sure that everything there is legally redistributable?

 from hutch... (https://masm32.com/board/index.php?msg=13796)

unicode UTF-16 string from windows nuclear
(https://i.postimg.cc/cvJGt0fD/Project001.gif) (https://postimg.cc/cvJGt0fD)

Hi bomz,
As you are currently on line, you NEED to address the concerns expressed by 'Biterider' in his PM to yourself as well as the concerns expressed in this 'thread' concerning the legality of your postings and Satisfy Us that everything is A-OK and 'above board' !!
ASAP !! OK ??
Stewart
The MASM32 Admin' team

grub4dos, bootice, microsoft virtual pc free software. winpe free for use

https://github.com/chenall/grub4dos/releases (https://github.com/chenall/grub4dos/releases)

Well, OK it is understood that;
grub4dos-0.4.6a-2022-01-18.7z
BOOTICE.1.3.2.7z
Microsoft Virtual PC 2007
Are Free and freely Re-Distributable,...
but that does not address the other files you have provided 'links' for.

Some degree of description and function etc would be appreciated.
Thanking you in advance  :smiley:

Quote from: stoo23 on March 10, 2025, 01:09:59 PMWell, OK it is understood that;
grub4dos-0.4.6a-2022-01-18.7z
BOOTICE.1.3.2.7z
Microsoft Virtual PC 2007
Are Free and freely Re-Distributable,...

The preferred method of transfer should be links to the official sites where any of those can be downloaded from, imho, and not from a users personal Google drive or file hosting account.

While bomz intentions may well have been good and all of that:
once in a users hands, we never know for certain if any of those have been manipulated in some fashion before redistribution outside of normal channels, i.e., from the original source, usually an official website.
If bomz used any special settings or configurations he can share that, without the need to share his whole development environment.

Just some food for thought.

Good point.

Quote from: zedd151 on March 10, 2025, 03:00:52 PMThe preferred method of transfer should be links to the official sites where any of those can be downloaded from, imho, and not from a users personal Google drive or file hosting account.

Yes, excellent point, and realistically I Should have made that distinction and have requested the same to have been applied.

To 'bomz' directly, may I suggest, it is perhaps a somewhat childish and petulant response to have simply deleted everything in response to what can only be described as a reasonable request.
Your input and interaction with the forum, is welcomed and had you not previously Raised the concerns of Hutch and other members over similar postings, all would have been fine.

I thank you for your observance and compliance to these simple rules in the future  :thumbsup: