Print Page - UTF8 performance

Title: UTF8 performance
Post by: Biterider on April 03, 2020, 12:41:17 AM

Hi
As you probably know, most communication between mobile devices uses UTF8 encoded strings. Because Windows uses wide strings (also known as UTF16) internally, many conversions need to be done. The two API routines in charge of the conversions are: MultiByteToWideChar and WideCharToMultiByte. They can perform various tasks like mapping characters from different code pages to Unicode Code Points or translate from one encoding to another. The former is useful if you are using code pages (8 bit characters), but if your application uses wide strings, these routines can encode or decode from UTF8 to UTF16.

Sometimes I got a better performance coding my own routines than using the Windows APIs. Therefore, I tried it here too. The two new routines are called WideToUTF8 and UTF8ToWide. I implemented a naive encoding described here https://de.wikipedia.org/wiki/UTF-8 (https://de.wikipedia.org/wiki/UTF-8) and here https://de.wikipedia.org/wiki/UTF-16 (https://de.wikipedia.org/wiki/UTF-16).

I also wrote a performance test using only the MASM32 framework and was really surprised how well the Windows routines performed. I spent some time tweaking my routines on my main machine and sometimes I got a 30% increase in performance in some Unicode regions (outside the BMP) and -10% in others. When I went to an older machine with the same code, I got a 30% general drop in performance.

My conclusion is that the MS routines are not bad at all. :icon_idea:

I post my code in case someone wants to try it out.

Biterider

Title: Re: UTF8 performance
Post by: LiaoMi on April 03, 2020, 01:47:54 AM

Hi Biterider,

i7-4810mq
-----------------------------------------------------------
Preparing the test...
RANGE = 0
Testing WideToUTF8...
SUCCEEDED WideToUTF8: 406 ticks, WideCharToMultiByte: 438 ticks = 7% faster
SUCCEEDED WideToUTF8: 390 ticks, WideCharToMultiByte: 438 ticks = 10% faster
SUCCEEDED WideToUTF8: 390 ticks, WideCharToMultiByte: 453 ticks = 13% faster
SUCCEEDED WideToUTF8: 391 ticks, WideCharToMultiByte: 438 ticks = 10% faster
SUCCEEDED WideToUTF8: 390 ticks, WideCharToMultiByte: 438 ticks = 10% faster
Ready

Testing UTF8ToWide...
SUCCEEDED UTF8ToWide: 609 ticks, MultiByteToWideChar: 688 ticks = 11% faster
SUCCEEDED UTF8ToWide: 578 ticks, MultiByteToWideChar: 687 ticks = 15% faster
SUCCEEDED UTF8ToWide: 578 ticks, MultiByteToWideChar: 672 ticks = 13% faster
SUCCEEDED UTF8ToWide: 594 ticks, MultiByteToWideChar: 672 ticks = 11% faster
SUCCEEDED UTF8ToWide: 578 ticks, MultiByteToWideChar: 672 ticks = 13% faster
Ready
-----------------------------------------------------------
Preparing the test...
RANGE = 1
Testing WideToUTF8...
SUCCEEDED WideToUTF8: 1406 ticks, WideCharToMultiByte: 1859 ticks = 24% faster
SUCCEEDED WideToUTF8: 1375 ticks, WideCharToMultiByte: 1875 ticks = 26% faster
SUCCEEDED WideToUTF8: 1438 ticks, WideCharToMultiByte: 1812 ticks = 20% faster
SUCCEEDED WideToUTF8: 1360 ticks, WideCharToMultiByte: 1828 ticks = 25% faster
SUCCEEDED WideToUTF8: 1375 ticks, WideCharToMultiByte: 1859 ticks = 26% faster
Ready

Testing UTF8ToWide...
SUCCEEDED UTF8ToWide: 2735 ticks, MultiByteToWideChar: 2797 ticks = 2% faster
SUCCEEDED UTF8ToWide: 2734 ticks, MultiByteToWideChar: 2703 ticks = -1% faster
SUCCEEDED UTF8ToWide: 2656 ticks, MultiByteToWideChar: 2735 ticks = 2% faster
SUCCEEDED UTF8ToWide: 2734 ticks, MultiByteToWideChar: 2734 ticks = 0% faster
SUCCEEDED UTF8ToWide: 2750 ticks, MultiByteToWideChar: 2766 ticks = 0% faster
Ready
-----------------------------------------------------------
Preparing the test...
RANGE = 2
Testing WideToUTF8...
SUCCEEDED WideToUTF8: 672 ticks, WideCharToMultiByte: 859 ticks = 21% faster
SUCCEEDED WideToUTF8: 657 ticks, WideCharToMultiByte: 875 ticks = 24% faster
SUCCEEDED WideToUTF8: 718 ticks, WideCharToMultiByte: 860 ticks = 16% faster
SUCCEEDED WideToUTF8: 687 ticks, WideCharToMultiByte: 875 ticks = 21% faster
SUCCEEDED WideToUTF8: 672 ticks, WideCharToMultiByte: 844 ticks = 20% faster
Ready

Testing UTF8ToWide...
SUCCEEDED UTF8ToWide: 1390 ticks, MultiByteToWideChar: 1329 ticks = -4% faster
SUCCEEDED UTF8ToWide: 1390 ticks, MultiByteToWideChar: 1297 ticks = -7% faster
SUCCEEDED UTF8ToWide: 1344 ticks, MultiByteToWideChar: 1344 ticks = 0% faster
SUCCEEDED UTF8ToWide: 1359 ticks, MultiByteToWideChar: 1312 ticks = -3% faster
SUCCEEDED UTF8ToWide: 1391 ticks, MultiByteToWideChar: 1344 ticks = -3% faster
Ready
-----------------------------------------------------------
Preparing the test...
RANGE = 3
Testing WideToUTF8...
SUCCEEDED WideToUTF8: 1172 ticks, WideCharToMultiByte: 1125 ticks = -4% faster
SUCCEEDED WideToUTF8: 1172 ticks, WideCharToMultiByte: 1109 ticks = -5% faster
SUCCEEDED WideToUTF8: 1187 ticks, WideCharToMultiByte: 1094 ticks = -8% faster
SUCCEEDED WideToUTF8: 1188 ticks, WideCharToMultiByte: 1109 ticks = -7% faster
SUCCEEDED WideToUTF8: 1156 ticks, WideCharToMultiByte: 1109 ticks = -4% faster
Ready

Testing UTF8ToWide...
SUCCEEDED UTF8ToWide: 1125 ticks, MultiByteToWideChar: 1375 ticks = 18% faster
SUCCEEDED UTF8ToWide: 1141 ticks, MultiByteToWideChar: 1375 ticks = 17% faster
SUCCEEDED UTF8ToWide: 1187 ticks, MultiByteToWideChar: 1375 ticks = 13% faster
SUCCEEDED UTF8ToWide: 1156 ticks, MultiByteToWideChar: 1391 ticks = 16% faster
SUCCEEDED UTF8ToWide: 1203 ticks, MultiByteToWideChar: 1359 ticks = 11% faster
Ready
Press any key to continue ...

Title: Re: UTF8 performance
Post by: HSE on April 03, 2020, 02:03:21 AM

Hi Biterider!

In this very fast turtle, UTF8ToWide always better :thumbsup:

Regards. HSE

Title: Re: UTF8 performance
Post by: mineiro on April 03, 2020, 02:15:33 AM

hello sir Biterider, tested in linux

The MASM Forum

Projects => ObjAsm => Topic started by: Biterider on April 03, 2020, 12:41:17 AM