### Author Topic: SMAC : Compression with Arithmetic Coding+Context Mixing  (Read 19172 times)

#### Jean-Marie

• Regular Member
• Posts: 17
• Waiting for TASM 6.0
##### SMAC : Compression with Arithmetic Coding+Context Mixing
« on: November 29, 2013, 09:15:28 PM »
This is the (open)source of my compression engine, released under zlib license.
It uses a Statistical method (as opposed to Dictionary method like LZ77) named Arithmetic Coding, where the entire input file is converted to a (very) long decimal number between 0 & 1.
Each time new bits are read from the input, this will narrow down the interval with the effect of expanding the codified number. Most frequent symbols will narrow the interval with a smaller impact than less frequent ones, hence a gain of compression.
AC is paired off with Context Models from Order-6 to Order-1.
Suppose you have to compress an english text; statistically, the letter ‘u’ would appear with a global frequency (or probability) of 5%.  Now, let’s suppose we know the previous character was a ‘q’ : the probability would now jump to 99%.
Context is simply a terminology for that previous character. In this example, we would use an Order-1 context model to gather the statistics of the following symbol. 2 previous characters would be Order-2 and so on... Obviously, using numerous models will have a bad impact on speed, and a good impact on compression.
After having calculated a probability for each model, we need to mix them up, hence the term Context Mixing. Among all different methods, I used Logistic Mixing, which is praised as one of the most efficient.
For more details, I strongly urge you to read Mark Nelson’s tutorial, which was my inspiration for writing this software. Matt Mahoney’s ebook is also a solid reference, and gives detail of his Logistic Mixing.
http://marknelson.us/1991/02/01/arithmetic-coding-statistical-modeling-data-compression/
http://mattmahoney.net/dc/dce.html#Section_432

Feel free to use (at your own risks), and modify : it is already optimized for speed, but may be you can find some new tricks. FYI, I have already tried replacing FPU instructions with SSE2, but surprisingly this was *much* slower. I also tried running some routines in a separate thread, using a Spinloop, and struggling against Race condition & False-sharing, with bad results either.

One last thing : I’m a die-hard TASM user, so you’ll have to convert it first. Luckily, it’s in MASM mode, so I guess you just have to replace the call API by this stupid invoke instruction

#### qWord

• Member
• Posts: 1476
• The base type of a type is the type itself
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #1 on: November 29, 2013, 10:01:20 PM »
I have already tried replacing FPU instructions with SSE2, but surprisingly this was *much* slower.
What did you replace? Sounds more like  an buggy implementation.
If the precision of 53 bit is sufficient, you should replace all basic arithmetic (+-*/,sqrt) with SSE2+, because that removes the overhead for the FPU stack. Furthermore you might replace  log2, 2^x or whole formulas with polynomial approximations.

qWord
MREAL macros - when you need floating point arithmetic while assembling!

#### Jean-Marie

• Regular Member
• Posts: 17
• Waiting for TASM 6.0
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #2 on: November 29, 2013, 10:52:23 PM »
I made some experiments with UpdateModel, Stretch, and also Squash. SSE2 is good for executing parallel calculations, otherwise it seems to be outperformed by FPU. Note that with FPU, you don't have the overhead of using conversion instructions(cvtdq**).
I have already used a lookup table for log2 function. Replacing the whole Squash formula by a LUT could be done in theory, but that would need to be a huge one since p0 ranges between -16,999999999... to +15,999999999. And reducing the mantissa would hurt the compression...
Code: [Select]
`;Update Weights in the logistic domain, such as: ;Wi=Wi+(Learning Rate*((1-Symbol-(Squash/2^32))*Stretch i)) fstp Temp ;store Squash value in local data (float) pcmpeqd xmm1,xmm1 psrld xmm1,31 ;xmm1 low dword=1 sub dword ptr Temp+4,2000000H ;substract 32 from exponent field cmp MatchPrediction,2 ;MatchPrediction=2 if no match found subss xmm1,Symbol ;xmm1=1-Symbol movsd xmm3,Temp ;xmm3=Squash/2^32 (float 64 bits) cvtdq2pd xmm1,xmm1 ;xmm1=1-Symbol (float 64 bits) subsd xmm1,xmm3 ;xmm1=(1-Symbol)-(Squash/2^32) mulsd xmm1,@@LearningRate ;xmm1=(1-Symbol)-(Squash/2^32)*Learning Rate cvtsd2ss xmm1,xmm1 ;xmm1=xmm0 converted to 32 bits precision movapd xmm3,xmm1 mulss xmm1,Stretch6 addss xmm1,Weight6 movss Weight6,xmm1 movapd xmm1,xmm3 mulss xmm1,Stretch4 addss xmm1,Weight4 movss Weight4,xmm1 movapd xmm1,xmm3 mulss xmm1,Stretch3 addss xmm1,Weight3 movss Weight3,xmm1 movapd xmm1,xmm3 mulss xmm1,Stretch2 addss xmm1,Weight2 movss Weight2,xmm1 movapd xmm1,xmm3 mulss xmm1,Stretch1 addss xmm1,Weight1 movss Weight1,xmm1 je @@IF1 ;skip update of Match if model is off movapd xmm1,xmm3 mulss xmm1,StretchM addss xmm1,WeightM movss WeightM,xmm1`
Code: [Select]
`@@Stretch: movzx edx,word ptr [edx] ;edx=Bit history of Order-x table@@Stretch2: mov edx,[edi+edx*4] movzx eax,dx ;eax=frequency of bit 0 movd xmm1,eax shr edx,16 ;edx=frequency of bit 1 lea ecx,[eax+edx] ;ecx=n0+n1 cvtdq2ps xmm1,xmm1 ;frequency of bit 0 (float) mulss xmm1,[ecx*4+ebx] ;xmm1=bit 0 range=Frequency*(2^32/TableCount) movd eax,xmm1 ;eax=p0 (float) cvtps2pd xmm2,xmm1 ;xmm2=p0 (float 64 bits) mov edx,eax shr eax,23 ;eax=biased exponent sub eax,127 ;eax=exponent and edx,7FFFFFH ;edx=mantissa shr edx,9 ;edx=mantissa reduced to 14 bits movd xmm1,[edx*4+esi] ;xmm1=log2(mantissa) cvtsi2ss xmm3,eax ;xmm3=exponent (float) addss xmm3,xmm1 ;xmm3=exp+log2(mantissa)=log2(p0) movsd xmm1, @@TwoPower32 ;xmm1=2^32 (float 64 bits) subsd xmm1,xmm2 ;xmm1=2^32-p0 cvtsd2ss xmm1,xmm1 ;convert xmm1 to float 32 bits movd eax,xmm1 ;eax=2^32-p0 (float) mov edx,eax shr eax,23 ;eax=biased exponent sub eax,127 ;eax=exponent and edx,7FFFFFH ;edx=mantissa shr edx,9 ;edx=mantissa reduced to 14 bits movd xmm1,[edx*4+esi] ;xmm1=log2(mantissa) cvtsi2ss xmm2,eax ;xmm2=exponent (float) addss xmm2,xmm1 ;xmm2=exp+log2(mantissa)=log2(2^32-p0) subss xmm3,xmm2 ;xmm3=log2(p0)-log2(2^32-p0) movss Temp,xmm3 fld dword ptr Temp ;store result in FPU stack ret`

#### jj2007

• Member
• Posts: 11450
• Assembler is fun ;-)
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #3 on: November 30, 2013, 12:20:59 AM »
I have translated the Tasm code to Masm syntax, but under Win7-32 it chokes in Flush_encoder (and that could obviously be a "translation error"):
deb 4, "before sub edi, OutputBuffer", x:edi, x:OutputBuffer, x:WriteBufferEnd
sub edi, OutputBuffer        ;last write of Output buffer to Destination file
lea eax, Temp
deb 4, "WriteFile in", @@TargetHandle, x:OutputBuffer, x:edi, eax
invoke        WriteFile, @@TargetHandle, OutputBuffer, edi, eax, NULL        ; edi is nNumberOfBytesToWrite
deb 4, "WriteFile out", \$Err\$()

Output:
before sub edi, OutputBuffer
x:edi           012A1026
x:OutputBuffer  112B0020
x:WriteBufferEnd        212B0020

WriteFile in
@@TargetHandle  152
x:OutputBuffer  112B0020
x:edi           EFFF1006     <<<<<<<<<<< edi is negative
eax             1245036

The official error message ("Out of Memory : file was too large to be processed") is misleading.
Testfile was \Masm32\include\masm32rt.inc, 3217 bytes.

Source attached, as plain Masm32; the *.asc file is RTF, opens e.g. in \Masm32\MasmBasic\RichMasm.exe for those who have MasmBasic installed.

EDIT: Fixed a pop reg reg translation bug, but still the same error.

#### qWord

• Member
• Posts: 1476
• The base type of a type is the type itself
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #4 on: November 30, 2013, 03:40:52 AM »
SSE2 is good for executing parallel calculations, otherwise it seems to be outperformed by FPU
surely not! For the basic arithmetic you will commonly get at least the same speed as with the FPU.

Note that with FPU, you don't have the overhead of using conversion instructions(cvtdq**)
That is no overhead - it is required and the FPU must do the same operation for FILD[P]/FIST[P]. For your SSE2 code I see the problem that there are several conversions that could be omitted.

Replacing the whole Squash formula by a LUT could be done in theory, but that would need to be a huge one since p0 ranges between -16,999999999... to +15,999999999.
What precision is required?
I've quickly worked out the following approximation (f(x) = 232/(1 + 2-x)), which divides the range -17 to 16 into 0.5 intervals. For each of them a polynomial (5th degree, mini-max approximation) is used. The |relative error| is  below 1.5*10-9:
Code: [Select]
`; xmm0 = input valuemovsd xmm1,xmm0addsd xmm0,sd_17addsd xmm0,xmm0cvttsd2si ecx,xmm0imul ecx,48lea ecx,tbl1[ecx]movsd xmm0,REAL8 ptr [ecx+0*8]mulsd xmm0,xmm1addsd xmm0,REAL8 ptr [ecx+1*8]mulsd xmm0,xmm1addsd xmm0,REAL8 ptr [ecx+2*8]mulsd xmm0,xmm1addsd xmm0,REAL8 ptr [ecx+3*8]mulsd xmm0,xmm1addsd xmm0,REAL8 ptr [ecx+4*8]mulsd xmm0,xmm1addsd xmm0,REAL8 ptr [ecx+5*8]; result in xmm0`the table:
Code: [Select]
` sd_17 REAL8 17.0 align 16; |error| <= 1.38272E-9; N = 66; degree = 5; table size = 3168.; step = 1/2 tbl1 LABEL REAL8    REAl8 51.88733919031596483941716    REAl8 4720.870462613202452233068    REAl8 172884.6360901519360640412    REAl8 3.188223395760284100183061E6    REAl8 2.963746272042194298627888E7    REAl8 1.112360695861580167739344E8    REAl8 73.37093794320208307773415    REAl8 6492.119208327092115313642    REAl8 231301.0872467358935065903    REAl8 4.151572985509914932599881E6    REAl8 3.758121739160534963936367E7    REAl8 1.374390625255227249973363E8    REAl8 103.7444950360918885549799    REAl8 8920.381139496640486548687    REAl8 308957.9720557864011553151    REAl8 5.393393990647069978685116E6    REAl8 4.751083833688111095777013E7    REAl8 1.691998295938006180216526E8    REAl8 146.6815143865613352699856    REAl8 12245.70757455918026048346    REAl8 411978.2806894580767835712    REAl8 6.989301161712469405214701E6    REAl8 5.987283593461696080019291E7    REAl8 2.075048719860747448085957E8    REAl8 207.3682817485600260967559    REAl8 16793.97732876753478507681    REAl8 548338.0852446730741007231    REAl8 9.033506521324976763685423E6    REAl8 7.519643238579234335081641E7    REAl8 2.534548108734618545114674E8    REAl8 293.1216536307773869953894    REAl8 23006.52660505454405322288    REAl8 728381.9784360021274601921    REAl8 1.164258358993160271096278E7    REAl8 9.410224813898590893585648E7    REAl8 3.082565013308120145304222E8    REAl8 414.2539305849464936082024    REAl8 31479.34026947846602799544    REAl8 965457.9370602551631181308    REAl8 1.495961451202408465767186E7    REAl8 1.173089653982246607197822E8    REAl8 3.732051486830684789845497E8    REAl8 585.2783733325399634026721    REAl8 43014.40949053542262511964    REAl8 1.276684556244585587565104E6    REAl8 1.915853681742603061636134E7    REAl8 1.456361596943700486275529E8    REAl8 4.496527615689425763894538E8    REAl8 826.5789317166141485281632    REAl8 58686.17363649724569071519    REAl8 1.683853518863027986762974E6    REAl8 2.444833048618222039159124E7    REAl8 1.800006682851481673533409E8    REAl8 5.389584065532640731318793E8    REAl8 1166.701540620524424160062    REAl8 79925.91362738045546319615    REAl8 2.214448514969364018692124E6    REAl8 3.107640588811522044036193E7    REAl8 2.214027446782516076500369E8    REAl8 6.424147749570835816767655E8    REAl8 1645.454949384501447527862    REAl8 108626.0494541723485621117    REAl8 2.902719901485979087662182E6    REAl8 3.933011677250708813703721E7    REAl8 2.708967526917784851648481E8    REAl8 7.611446113728958057706480E8    REAl8 2318.018439277405107695768    REAl8 147263.5525706455660606572    REAl8 3.790677845570720009593682E6    REAl8 4.953465629544336855878542E7    REAl8 3.295392107360258782133333E8    REAl8 8.959597617768673797197731E8    REAl8 3260.200452275777993043329    REAl8 199035.3156538426108962666    REAl8 4.928732482515329218100044E6    REAl8 6.204460080945446003032485E7    REAl8 3.983045899138597431275463E8    REAl8 1.047175340385311826795597E9    REAl8 4574.786887985574618064181    REAl8 267985.0671166693348622539    REAl8 6.375482039357043602296603E6    REAl8 7.722493099671766991332702E7    REAl8 4.779565315896772268256312E8    REAl8 1.214372485205343391588581E9    REAl8 6398.377281369057850132229    REAl8 359075.5502263001980307799    REAl8 8.195776627521903805948546E6    REAl8 9.541538074218751523172678E7    REAl8 5.688595641485783918309568E8    REAl8 1.396106632654373763609108E9    REAl8 8906.868411625623359851656    REAl8 478111.5585657789651670472    REAl8 1.045559703676712106312419E7    REAl8 1.168694372024877518125471E8    REAl8 6.707151683369115095043724E8    REAl8 1.589566334450725218731593E9    REAl8 12315.11751038764749208423    REAl8 631333.7374995353116777207    REAl8 1.321140989842440560380851E7    REAl8 1.416565384021648426046242E8    REAl8 7.822087291184103918278085E8    REAl8 1.790203745507358979575692E9    REAl8 16861.04993647702652692920    REAl8 824358.0188788822263523139    REAl8 1.649046073334505468534142E7    REAl8 1.695140129565972815461480E8    REAl8 9.005654216704815700431120E8    REAl8 1.991386813593067849564320E9    REAl8 22754.46441713161584215514    REAl8 1.059906537737560762124723E6    REAl8 2.025708483621198742030132E7    REAl8 1.996366396615991261446319E8    REAl8 1.021041863550817982651474E9    REAl8 2.184170452641999693025212E9    REAl8 30052.94003938008067652104    REAl8 1.333453789840112773980128E6    REAl8 2.435916941937797588827382E7    REAl8 2.304017891703084958276696E8    REAl8 1.136439160889121268842937E9    REAl8 2.357353101661176436114812E9    REAl8 38398.32828116109481622022    REAl8 1.625559490940538651663771E6    REAl8 2.845009739625804371902311E7    REAl8 2.590568889751042406970710E8    REAl8 1.236826787688546966331589E9    REAl8 2.498070046735167137716712E9    REAl8 46508.18760778884594428123    REAl8 1.889552700529974441505065E6    REAl8 3.188867359407945289359876E7    REAl8 2.814585690098661576991069E8    REAl8 1.309822612862406588867451E9    REAl8 2.593244476008529728633210E9    REAl8 51286.48680556550798215726    REAl8 2.034171855556533214405132E6    REAl8 3.364002008981092958762780E7    REAl8 2.920662452804249836281330E8    REAl8 1.341956881166352482231491E9    REAl8 2.632194212836124515045006E9    REAl8 46458.87558548222189352842    REAl8 1.904246973569067231921234E6    REAl8 3.224128809169131925320706E7    REAl8 2.845366972054214218500149E8    REAl8 1.321689598615280224161707E9    REAl8 2.610371726286179527153147E9    REAl8 20975.51806748501375713108    REAl8 1.272504738849985470093961E6    REAl8 2.597353007120837620438940E7    REAl8 2.534276454565097162970081E8    REAl8 1.244445288989806200988076E9    REAl8 2.533610123826259437033142E9    REAl8 -41665.44784579087591401305    REAl8 -128327.7645362013802850065    REAl8 1.343419421779237837243741E7    REAl8 1.972664551890367319252868E8    REAl8 1.118589292225542095616711E9    REAl8 2.420713940265105676429838E9    REAl8 -159925.2357718777121206428    REAl8 -2.482454180858725956371900E6    REAl8 -5.327422481589664148300558E6    REAl8 1.224372553089571028572369E8    REAl8 9.692293754178528354979665E8    REAl8 2.301356342160404183516524E9    REAl8 -340704.2751791445351200297    REAl8 -5.637361869299855401966814E6    REAl8 -2.737675127846776945036919E7    REAl8 4.529620595971246012769910E7    REAl8 8.341288379899892713131339E8    REAl8 2.206602137416587262624473E9    REAl8 -550688.4573738412971652942    REAl8 -8.792287556340922002923093E6    REAl8 -4.636726080472468121215988E7    REAl8 -1.194845141320483499575401E7    REAl8 7.477161208745740109585162E8    REAl8 2.154344969410214654677325E9    REAl8 -682084.2886640759045159987    REAl8 -1.046972974352329558032552E7    REAl8 -5.494779866854977968977399E7    REAl8 -3.393037845944035201999181E7    REAl8 7.195145890567937170329878E8    REAl8 2.139850740573633841417918E9    REAl8 -558162.3586669414578327902    REAl8 -9.300481840357780218319035E6    REAl8 -5.053236320407293140768169E7    REAl8 -2.558848389629871480683629E7    REAl8 7.273992081693988067212635E8    REAl8 2.142833418771404723546764E9    REAl8 -48522.57832244910139469034    REAl8 -5.551935872604515445945657E6    REAl8 -3.943509185975672954158751E7    REAl8 -9.058245938290568166120203E6    REAl8 7.397888127157848671876201E8    REAl8 2.146571143434234695393244E9    REAl8 726422.4115865083010185224    REAl8 -1.686984749507153275137732E6    REAl8 -3.161483717689012300886614E7    REAl8 -1.037726009165054009298133E6    REAl8 7.439547125925731762081104E8    REAl8 2.147446772140937142388343E9    REAl8 1.332860634151411311832797E6    REAl8 -74937.92759159643898995843    REAl8 -2.982077072597132270618378E7    REAl8 -2845.635313607631774856469    REAl8 7.442609865909374549425550E8    REAl8 2.147483647014656632493609E9    REAl8 1.335383913415405996515172E6    REAl8 71760.75132826929181392017    REAl8 -2.981936752714387582471325E7    REAl8 2591.855502032223925773167    REAl8 7.442610027424416123681440E8    REAl8 2.147483648819214358475640E9    REAl8 732051.9171355465161504821    REAl8 1.665853223774519926133289E6    REAl8 -3.158354830562147849911415E7    REAl8 1.014896776518033605177229E6    REAl8 7.439629167360230450583053E8    REAl8 2.147519362430839555255322E9    REAl8 -43516.16926320761131628149    REAl8 5.520633305127000158802576E6    REAl8 -3.935719525108734579673865E7    REAl8 8.961814580893349163105159E6    REAl8 7.398481947822541111526046E8    REAl8 2.148381601094616583203359E9    REAl8 -555921.2122245761574794132    REAl8 9.280871856349502145332422E6    REAl8 -5.046390348689352598440538E7    REAl8 2.546929239223269358817666E7    REAl8 7.275026990367835896043112E8    REAl8 2.152098026967556942059480E9    REAl8 -682338.0427011109666668739    REAl8 1.047259356078063484176302E7    REAl8 -5.496070718958433717490771E7    REAl8 3.395942613924233865750694E7    REAl8 7.194819568827038098825054E8    REAl8 2.155131196298104028886998E9    REAl8 -552102.7944867689552141823    REAl8 8.811748344096884160537542E6    REAl8 -4.647425982677673214475332E7    REAl8 1.224229716940814689065877E7    REAl8 7.473130531668233332682037E8    REAl8 2.140843251532804229212566E9    REAl8 -342194.4887024370594115037    REAl8 5.661589742560789332911280E6    REAl8 -2.753419347450882094913362E7    REAl8 -4.478502439885995820700179E7    REAl8 8.332996046052223682557839E8    REAl8 2.088902828232224080359230E9    REAl8 -161013.7461251784840607237    REAl8 2.502871775334219798084822E6    REAl8 -5.480529635610485900873517E6    REAl8 -1.218635157298784007561354E8    REAl8 9.681549848455054270748302E8    REAl8 1.994415271074422430609307E9    REAl8 -42299.66082296048907950085    REAl8 141809.1455379105287653525    REAl8 1.331961498347535586151603E7    REAl8 -1.967797573339104896614340E8    REAl8 1.117556067695612863684682E9    REAl8 1.875130359761998188226793E9    REAl8 20679.28972340881200391904    REAl8 -1.265467531954631347834732E6    REAl8 2.590668279324195458102168E7    REAl8 -2.531102602697656277104965E8    REAl8 1.243692091343737040658066E9    REAl8 1.762071899729681598408167E9    REAl8 46365.72774184264862518188    REAl8 -1.901801537210444875673743E6    REAl8 3.221561507717393112474085E7    REAl8 -2.844019734430491972025887E8    REAl8 1.321336205333986142051703E9    REAl8 1.684966258168738115434647E9    REAl8 51296.26906479652243893437    REAl8 -2.034453460798958838014325E6    REAl8 3.364326199808319996811902E7    REAl8 -2.920849017275537820841159E8    REAl8 1.342010550377863576164611E9    REAl8 1.662711341351493053323293E9    REAl8 46559.93123999386323884351    REAl8 -1.891170333934258053804151E6    REAl8 3.190889803596906165872110E7    REAl8 -2.815849716529626884455225E8    REAl8 1.310217541716875540965410E9    REAl8 1.701229356972405374298253E9    REAl8 38460.12086416241761948303    REAl8 -1.627645650398769250723169E6    REAl8 2.847826463335892321484685E7    REAl8 -2.592470128012970606378521E8    REAl8 1.237468328576388340990004E9    REAl8 1.796031489127965865506578E9    REAl8 30110.28394493323552923772    REAl8 -1.335533083132631871219997E6    REAl8 2.438932305858293550992657E7    REAl8 -2.306203987449487887401174E8    REAl8 1.137231487150732082962270E9    REAl8 1.936465685588529349550436E9    REAl8 22802.22654190181919955177    REAl8 -1.061757786076044164071924E6    REAl8 2.028578268031018092051072E7    REAl8 -1.998590461157658387397019E8    REAl8 1.021903569309950097004185E9    REAl8 2.109461558065995587095947E9    REAl8 16898.57206835948319166550    REAl8 -825906.1665794047656624242    REAl8 1.651600817748027651538321E7    REAl8 -1.697247799926236575789155E8    REAl8 9.014347384432682621387512E8    REAl8 2.302146434894433289603890E9    REAl8 12343.57580198898116097439    REAl8 -632579.0576846077662020517    REAl8 1.323320550391246943614610E7    REAl8 -1.418472523900361965095591E8    REAl8 7.830430283356968093851157E8    REAl8 2.503303806228949611480746E9    REAl8 8927.973064356175892775077    REAl8 -479087.8475410828410687272    REAl8 1.047366041289495487078952E7    REAl8 -1.170365269584774042270141E8    REAl8 6.714879042000377509580123E8    REAl8 2.703971631145601320235713E9    REAl8 6413.797957054993430475087    REAl8 -359827.4527423775650911503    REAl8 8.210440342052674839141764E6    REAl8 -9.555835596385718941175884E7    REAl8 5.695565305924893023512178E8    REAl8 2.897501764135767872771733E9    REAl8 4585.942033355371218497164    REAl8 -268556.8725734998419099484    REAl8 6.387205317683773997301802E6    REAl8 -7.734509864859499090204063E7    REAl8 4.785723657594800518327158E8    REAl8 3.079332495988220877147896E9    REAl8 3268.214712815484995751706    REAl8 -199466.1568980733965572464    REAl8 4.937996550565582048580656E6    REAl8 -6.214419335992694732549847E7    REAl8 3.988398842101365603505676E8    REAl8 3.246641184370984739532365E9    REAl8 2323.748866690648900071246    REAl8 -147585.9424742034164932004    REAl8 3.797932371701946848766331E6    REAl8 -4.961627315321717607565446E7    REAl8 3.299982965658639909564438E8    REAl8 3.397974675117164069626992E9    REAl8 1649.538824089230525077467    REAl8 -108866.0150823789643863096    REAl8 2.908359667697785142307522E6    REAl8 -3.939638702112656562451389E7    REAl8 2.712860859716973326815121E8    REAl8 3.532907813611478820497266E9    REAl8 1169.605238950241866908562    REAl8 -80103.79214136446242493789    REAl8 2.218806973176802228917185E6    REAl8 -3.112979954677305528516105E7    REAl8 2.217297794048506523087861E8    REAl8 3.651751331191003852761445E9    REAl8 828.6401511981534130941397    REAl8 -58817.59552867366429668745    REAl8 1.687205104539020066107648E6    REAl8 -2.449106532548697109630451E7    REAl8 1.802731033217287024241470E8    REAl8 3.755314212381145710209216E9    REAl8 586.7398776587289522026681    REAl8 -43111.24772569231545208833    REAl8 1.279251015085520036800340E6    REAl8 -1.919254413885470255252510E7    REAl8 1.458614597137765221811009E8    REAl8 3.844717511935121969394817E9    REAl8 415.2893728951092056474509    REAl8 -31550.53654240797193383960    REAl8 967416.0181237519759476997    REAl8 -1.498653953855606180656706E7    REAl8 1.174940770145503320649976E8    REAl8 3.921253105746302081620290E9    REAl8 293.8548238435451995418708    REAl8 -23058.77178899783276696193    REAl8 729871.1029238171823416873    REAl8 -1.166380474650067195995057E7    REAl8 9.425345113327178281336295E7    REAl8 3.986279876291964299980339E9    REAl8 207.8872125527905035088649    REAl8 -16832.25329239257675326877    REAl8 549467.3277581640143523852    REAl8 -9.050163748985590180379262E6    REAl8 7.531928167114279138649217E7    REAl8 4.041150086165972192611281E9    REAl8 147.0487045383223535089716    REAl8 -12273.70923165460640302243    REAl8 412832.4065115245922415969    REAl8 -7.002327291189103659302461E6    REAl8 5.997216225499573116012021E7    REAl8 4.087159482559950174789489E9    REAl8 104.0042630389798698669918    REAl8 -8940.840280572152975131376    REAl8 309602.4906630373519650869    REAl8 -5.403545715874895932158144E6    REAl8 4.759078504351004841642775E7    REAl8 4.125515636319303188802148E9`
MREAL macros - when you need floating point arithmetic while assembling!

#### TWell

• Member
• Posts: 748
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #5 on: November 30, 2013, 04:32:27 AM »
What is that error ?
Code: [Select]
`SMAC.asm(165):  error A2070:invalid instruction operandsMicrosoft (R) Macro Assembler Version 10.00.40219.01Copyright (C) Microsoft Corporation.  All rights reserved.`
Code: [Select]
`DisplayHelp: invoke WriteFile,hStdOut,offset Helpmsg,offset Waitmsg-offset Helpmsg-1,offset Dummy,NULL`

#### dedndave

• Member
• Posts: 8829
• Still using Abacus 2.0
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #6 on: November 30, 2013, 04:51:24 AM »
i rarely see that error on INVOKE lines, but you might try this...
Code: [Select]
`TmpOperand = offset Waitmsg-offset Helpmsg-1 invoke WriteFile,hStdOut,offset Helpmsg,TmpOperand,offset Dummy,NULL`
not really great form
if WaitMsg gets moved in the data section, the program will break

#### TWell

• Member
• Posts: 748
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #7 on: November 30, 2013, 05:11:47 AM »
I compiled that file with POAsm, so there must be something i missed.

Jochen, how you manged to do that with ml.exe?

EDIT: JWasm compiled this file

#### jj2007

• Member
• Posts: 11450
• Assembler is fun ;-)
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #8 on: November 30, 2013, 05:52:44 AM »
I compiled that file with POAsm, so there must be something i missed.

Jochen, how you manged to do that with ml.exe?

EDIT: JWasm compiled this file

I translated it to Masm syntax, see attachment to reply #3.

Astonishing that JWasm assembles your code ::)
However, it still chokes:
Out of Memory : file was too large to be processed

#### TWell

• Member
• Posts: 748
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #9 on: November 30, 2013, 05:56:21 AM »
Code: [Select]
`JWasm v2.10, Apr 20 2013, Masm-compatible assembler.Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.Source code is available under the Sybase Open Watcom Public License.SMAC.asm: 1651 lines, 4 passes, 16 ms, 0 warnings, 0 errors`BTW: this was my exercise to convert it for POAsm.exe, as i'm not an assembler programmer  ;).
I could be glad if pro's correct my mistakes.

#### jj2007

• Member
• Posts: 11450
• Assembler is fun ;-)
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #10 on: November 30, 2013, 06:16:48 AM »
TWell, good job :t
Can you compress files? With your exe, I still get the "file too large" error, now on Win XP SP3.
Same for Jean-Marie's executable...
@echo off
SMAC.exe /c win32.inc
pause >nul

#### TWell

• Member
• Posts: 748
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #11 on: November 30, 2013, 06:32:41 AM »
Code: [Select]
`smac /c SMAC.asm.bakPlease Wait...Finished!smac /d SMAC.asm.bak.smaPlease Wait...Finished!`

#### jj2007

• Member
• Posts: 11450
• Assembler is fun ;-)
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #12 on: November 30, 2013, 02:13:37 PM »
Mysterious ::)

I use exactly the same commandline, and both in WinXP SP3 and Win7-32, I get that error message; which is misleading on Win7-32 because it's WriteFile reporting a bad address, and misleading on WinXP because the processed file is tiny.

Who else has tested it? Attached a diagnostic version, which yields for me
- not enough memory for BitHistory on XP (where I have 2 GB of RAM)

#### Jean-Marie

• Regular Member
• Posts: 17
• Waiting for TASM 6.0
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #13 on: November 30, 2013, 07:32:13 PM »
@qWord : Interesting : I'm gonna test that. Regarding precision, the bigger the better of course, but I don't mind hurting the compression ratio (too much) for the benefit of speed. Thanks!

#### Jean-Marie

• Regular Member
• Posts: 17
• Waiting for TASM 6.0
##### Re: SMAC : Compression with Arithmetic Coding+Context Mixing
« Reply #14 on: November 30, 2013, 07:53:09 PM »
@JJ2007: looks like you restored the registers in the wrong order :
Code: [Select]
`@@Error: deb 4, "@@Error", \$Err\$() pop ebx ;  pop edi esi edx ecx ebx pop ecx pop edx pop esi pop edi ret`