New line tokeniser.

[TouEnMasm]

I have posted the text routines in used in the translater (enough improved)
They are a full soluce to work with text and lines numbers.
http://masm32.com/board/index.php?topic=3715.msg39270#msg39270

[Siekmanski]

Win 8.1 64bit

Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX)
Hutch   Yves    Recall
4155    847     828 us
1823    786     737 us
1772    756     769 us
1796    776     783 us
1767    782     733 us
1762    762     760 us
1764    775     728 us
1770    773     760 us
1771    774     734 us
1772    767     731 us
1788    773     734 us
1778    779     728 us
1794    804     740 us

[jj2007]

Thanks, Marinus :icon14:

Windows XP,access violation bad adress

004014ae 0fb60a          movzx   ecx,byte ptr [edx]         ds:0023:00000000=??

@Hutch:
It's line 194ff:
    add edx, 1
    movzx ecx, BYTE PTR [edx]
    cmp ecx, 13

... but no worries, your code is ok. The dump says edx is zero, i.e. no source, and Yves launched it from a temp C: folder, while his Windows.inc is probably on another drive - see InputFile("\Masm32\include\Windows.inc"). My fault, my code should shout foul if the file isn't loaded.

@Yves:
Re text routines: Do you seriously expect that somebody starts searching for the right one in that archive? Indicate clearly which is the line tokeniser (if you have one), or post it here.

New version attached - it will complain if there is no windows.inc in the current drive. Note that "Yves" represents the algo CompteurLignes, which performs only what Hutch's subroutine get_lcnt does, i.e. no array returned.

Code Select Expand

Intel(R) Core(TM) i5-2450M CPU @ 2.50GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX)
Testing 47 MB in tmp_test.txt

Hutch   Yves    Recall
117     53      45 ms
110     52      49 ms
109     52      49 ms
110     52      52 ms
115     52      49 ms
110     51      49 ms
110     51      49 ms
109     52      49 ms
110     52      51 ms
111     51      49 ms
111     52      49 ms
110     51      50 ms
109     52      49 ms


Results Recall:
echo ------------------------------------------
echo WARNING Duplicate include file windows.inc
echo ------------------------------------------
ENDIF


Results Hutch:
echo ------------------------------------------
echo WARNING Duplicate include file windows.inc
echo ------------------------------------------
ENDIF


(No results for Yves - line count only)
Thats all folks, press a key to exit...

[Siekmanski]

Code Select Expand

Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, S
SE4.2, AVX)
Testing 0 MB in tmp_test.txt

Hutch   Yves    Recall
113     44      49 ms
99      47      46 ms
96      45      45 ms
95      47      46 ms
96      45      46 ms
96      45      46 ms
96      46      46 ms
96      45      46 ms
96      47      48 ms
98      45      46 ms
96      45      46 ms
96      45      46 ms
96      45      46 ms


Results Recall:
echo ------------------------------------------
echo WARNING Duplicate include file windows.inc
echo ------------------------------------------
ENDIF


Results Hutch:
echo ------------------------------------------
echo WARNING Duplicate include file windows.inc
echo ------------------------------------------
ENDIF


(No results for Yves - line count only)
Thats all folks, press a key to exit...

[TouEnMasm]

Better result in the disk with masm32

Intel(R) Celeron(R) CPU 2.80GHz (MMX, SSE, SSE2, SSE3)
Hutch   Yves    Recall
85543   3136    3059 us
5276    3099    5293 us
5623    3126    3291 us
5328    3098    3514 us
5331    3057    3272 us
5571    3107    3231 us
5237    3141    3294 us
5314    3108    3365 us
5503    3060    3260 us
5250    3091    3228 us
5231    3168    3260 us
5274    3096    3300 us
5298    3048    3688 us


Results Hutch:
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
; It is not recomended that WINDOWS.INC be modified but if you need to add
; equates or structures to WINDOWS.INC, do not write anything after the
; following conditional assembly directive that display the duplicate
; warning or it will be duplicated if the file is included more than once.
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««

ELSE
echo ------------------------------------------
echo WARNING Duplicate include file windows.inc
echo ------------------------------------------
ENDIF

Thats all folks, press a key to exit...

The routines I have posted are not for test here (The line counter is in the test).

[Gunther]

Jochen,

the new version works fine under Windows7-64. Here are the results:

Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SS
E4.2, AVX)
Testing 47 MB in tmp_test.txt

Hutch   Yves    Recall
88      41      37 ms
92      43      43 ms
85      39      40 ms
84      39      40 ms
84      39      40 ms
84      39      40 ms
84      39      40 ms
84      39      39 ms
84      39      40 ms
84      39      40 ms
84      39      40 ms
85      39      40 ms
84      39      40 ms


Results Recall:
echo ------------------------------------------
echo WARNING Duplicate include file windows.inc
echo ------------------------------------------
ENDIF


Results Hutch:
echo ------------------------------------------
echo WARNING Duplicate include file windows.inc
echo ------------------------------------------
ENDIF


(No results for Yves - line count only)
Thats all folks, press a key to exit...

Gunther

[jj2007]

Thanks, Marinus & Gunther :t

The routines I have posted are not for test here

:P

[TouEnMasm]

For JJ,I see that you have not read the post,it's usefull sometimes:

The text routines have been tested in the translator since 2010

[jj2007]

Which text routines? Which translator? And why should anybody wish to dig (fouiller) into your archives, which obviously do not contain a tokeniser (otherwise you would have proudly posted it here)?

[TouEnMasm]

Sorry that google don't know what is a tokeniser.
What do the routines i have posted is an array with:
         A pointer on each line
         The size of each line

This allow to get a line by his number,without any scrutation.

[jj2007]

         A pointer on each line
         The size of each line

That sounds interesting. Now, if you reveal the name of the file and the name of the algo, I might give it a try.

[TouEnMasm]

It is here :Texte_Routines\lib\Texte_scrutTxt.asm
http://masm32.com/board/index.php?topic=3715.msg39270#msg39270

In the source file ,you find "Scrutationdeux" it save each pointer and size after the count of lines had been made and memory allocated.

[Gunther]

It is here :Texte_Routines\lib\Texte_scrutTxt.asm
http://masm32.com/board/index.php?topic=3715.msg39270#msg39270

In the source file ,you find "Scrutationdeux" it save each pointer and size after the count of lines had been made and memory allocated.

found it.  :t

Gunther

[jj2007]

In the source file ,you find "Scrutationdeux"

Line 387:

   sub eax,esi
   .if eax == 0
      jmp FindeMemoriseLigne      
   .endif

Ever heard of dead code and conditional jumps?

   sub eax,esi
   je FindeMemoriseLigne   

It doesn't assemble, and since the whole thing looks a bit slow, I guess I'll just switch on the TV and have a nice evening

[hutch--]

> I guess I'll just switch on the TV and have a nice evening

Sounds like you are doing it tough.  :P