How to Unit-Test Assembly

[LiaoMi]

Hello,

wondered if it was possible, found a discussion about it "What unit testing frameworks are available for x86 assembler?" - https://stackoverflow.com/questions/9529136/what-unit-testing-frameworks-are-available-for-x86-assembler
How to Unit-Test Assembly (NASM) - http://blog.code-cop.org/2015/08/how-to-unit-test-assembly.html
Something You May Not Know About the Macro in MASM https://www.codeproject.com/Articles/1080585/Something-You-May-Not-Know-About-the-Macro-in-MASM

[jj2007]

What is that 'unit testing framework' supposed to do? I only see some very basic macros for printing etc., nothing even remotely as useful as deb, NanoTimer or MemState.

The CodeProject article may be fine for a beginner in macro programming, but it shows mainly that only a real masochist would do MASM programming in Visual Crap.

[LiaoMi]

What is that 'unit testing framework' supposed to do? I only see some very basic macros for printing etc., nothing even remotely as useful as deb, NanoTimer or MemState.

The CodeProject article may be fine for a beginner in macro programming, but it shows mainly that only a real masochist would do MASM programming in Visual Crap.

Hi jj2007,

ideally, the technique should be able to use all the same possibilities as in a high-level language https://en.wikipedia.org/wiki/Unit_testing. This is similar to debug messages, but with more features.
More in the short example - https://www.youtube.com/watch?v=rW6LvPP4VvA (Automatic verification of input and output data.)
I gave only an example of tools that can be used, using a visual studio is optional  I think for assembly language is definitely not required.

Advantages
The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits.

Unit testing finds problems early in the development cycle. This includes both bugs in the programmer's implementation and flaws or missing parts of the specification for the unit. The process of writing a thorough set of tests forces the author to think through inputs, outputs, and error conditions, and thus more crisply define the unit's desired behavior. The cost of finding a bug before coding begins or when the code is first written is considerably lower than the cost of detecting, identifying, and correcting the bug later. Bugs in released code may also cause costly problems for the end-users of the software. Code can be impossible or difficult to unit test if poorly written, thus unit testing can force developers to structure functions and objects in better ways.

In test-driven development (TDD), which is frequently used in both extreme programming and scrum, unit tests are created before the code itself is written. When the tests pass, that code is considered complete. The same unit tests are run against that function frequently as the larger code base is developed either as the code is changed or via an automated process with the build. If the unit tests fail, it is considered to be a bug either in the changed code or the tests themselves. The unit tests then allow the location of the fault or failure to be easily traced. Since the unit tests alert the development team of the problem before handing the code off to testers or clients, potential problems are caught early in the development process.

Unit testing allows the programmer to refactor code or upgrade system libraries at a later date, and make sure the module still works correctly (e.g., in regression testing). The procedure is to write test cases for all functions and methods so that whenever a change causes a fault, it can be quickly identified. Unit tests detect changes which may break a design contract.

Unit testing may reduce uncertainty in the units themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier.[citation needed]

Unit testing provides a sort of living documentation of the system. Developers looking to learn what functionality is provided by a unit, and how to use it, can look at the unit tests to gain a basic understanding of the unit's interface (API).

Unit test cases embody characteristics that are critical to the success of the unit. These characteristics can indicate appropriate/inappropriate use of a unit as well as negative behaviors that are to be trapped by the unit. A unit test case, in and of itself, documents these critical characteristics, although many software development environments do not rely solely upon code to document the product in development.

When software is developed using a test-driven approach, the combination of writing the unit test to specify the interface plus the refactoring activities performed after the test has passed, may take the place of formal design. Each unit test can be seen as a design element specifying classes, methods, and observable behaviour.

[LiaoMi]

In simple words, unit testing is a macro function that tests another function, input and output parameters, environment variables and logical transitions according to previously known data  And then displays a table of test results. Writing another function to test the functions is too expensive, so unit testing should theoretically simplify this task. Therefore, the question is what methods can be used to implement such an approach in assembly language.

[jj2007]

I see. One example could be my gdi+ macro (used only internally, undocumented). I use it instead of invoke when calling GdiPlus functions, e.g. in

Code Select Expand

gdi+ GdipAddPathEllipse, mbsPath, [eax], [eax+4], [eax+8], [eax+12] ; (*path, REAL x, REAL y, REAL width, REAL height)

gdi has an assembly time variable which can be set to three states:
- release mode: do not create any code, just invoke the call
- debug, errors only: create code that checks if invoke GdiWhatever returned the expected value S_OK; bark only if it didn't
- debug, show all: create code that displays all GdiWhatever calls and their return value

For example, one of the MasmBasic templates returns this in "show all" mode:

Code Select Expand

** line 16, GdipCreatePen1                    Ok
** line 17, GdipCreatePen1                    Ok
** line 18, GdipCreatePen1                    Ok
** line 18, GdipCreateAdjustableArrowCap      Ok
** line 18, GdipSetPenCustomEndCap            Ok
** line 19, GdipCreatePen1                    Ok
** line 19, GdipSetPenStartCap                Ok
** line 19, GdipCreateAdjustableArrowCap      Ok
** line 19, GdipSetAdjustableArrowCapMiddleI  Ok
** line 19, GdipSetPenCustomEndCap            Ok
** line 21, GdipCreateSolidFill               Ok
** line 22, GdipCreateSolidFill               Ok
** line 23, GdipCreateSolidFill               Ok
** line 24, GdipCreateSolidFill               Ok
** line 25, GdipCreateSolidFill               Ok
** line 26, GdipCreateSolidFill               Ok
** line 27, GdipCreateSolidFill               Ok
** line 75, GdipCreateFromHDC                 Ok
** line 75, GdipSetSmoothingMode              Ok

Which means everything worked as expected (not that useful except that it may show the order of calls).

I must say that this macro was damn useful when developing the gdiplus stuff. I wrote it because manually checking why certain things didn't work was extremely difficult - we are talking graphics and WM_PAINT events here, debugging is almost impossible.

I use a similar assembly time variable for CoInvoke, with several switches:

Code Select Expand

CoInvokeCheck=0 ; 2+4+8+16 ; check if COM behaves well (adds lots of code)
; and 1 report errors. i.e. eax!=S_OK
; and 2 no error for eax==1 alias S_FALSE (often not an error)
; and 4 show all CoInvoke lines regardless of outcome
; and 8 write errors to CoInvokeLog.txt
; and 16 show output as hex

In short: Yes, such macros are very useful for testing and bug chasing, especially since you can switch off completely the generation of extra code once you think it's good for a release version :t

[LiaoMi]

I see. One example could be my gdi+ macro (used only internally, undocumented). I use it instead of invoke when calling GdiPlus functions, e.g. in

Code Select Expand

gdi+ GdipAddPathEllipse, mbsPath, [eax], [eax+4], [eax+8], [eax+12] ; (*path, REAL x, REAL y, REAL width, REAL height)

gdi has an assembly time variable which can be set to three states:
- release mode: do not create any code, just invoke the call
- debug, errors only: create code that checks if invoke GdiWhatever returned the expected value S_OK; bark only if it didn't
- debug, show all: create code that displays all GdiWhatever calls and their return value

For example, one of the MasmBasic templates returns this in "show all" mode:

Code Select Expand

** line 16, GdipCreatePen1                    Ok
** line 17, GdipCreatePen1                    Ok
** line 18, GdipCreatePen1                    Ok
** line 18, GdipCreateAdjustableArrowCap      Ok
** line 18, GdipSetPenCustomEndCap            Ok
** line 19, GdipCreatePen1                    Ok
** line 19, GdipSetPenStartCap                Ok
** line 19, GdipCreateAdjustableArrowCap      Ok
** line 19, GdipSetAdjustableArrowCapMiddleI  Ok
** line 19, GdipSetPenCustomEndCap            Ok
** line 21, GdipCreateSolidFill               Ok
** line 22, GdipCreateSolidFill               Ok
** line 23, GdipCreateSolidFill               Ok
** line 24, GdipCreateSolidFill               Ok
** line 25, GdipCreateSolidFill               Ok
** line 26, GdipCreateSolidFill               Ok
** line 27, GdipCreateSolidFill               Ok
** line 75, GdipCreateFromHDC                 Ok
** line 75, GdipSetSmoothingMode              Ok

Which means everything worked as expected (not that useful except that it may show the order of calls).

I must say that this macro was damn useful when developing the gdiplus stuff. I wrote it because manually checking why certain things didn't work was extremely difficult - we are talking graphics and WM_PAINT events here, debugging is almost impossible.

I use a similar assembly time variable for CoInvoke, with several switches:

Code Select Expand

CoInvokeCheck=0 ; 2+4+8+16 ; check if COM behaves well (adds lots of code)
; and 1 report errors. i.e. eax!=S_OK
; and 2 no error for eax==1 alias S_FALSE (often not an error)
; and 4 show all CoInvoke lines regardless of outcome
; and 8 write errors to CoInvokeLog.txt
; and 16 show output as hex

In short: Yes, such macros are very useful for testing and bug chasing, especially since you can switch off completely the generation of extra code once you think it's good for a release version :t

:t  :icon14: :icon14: :icon14: can you customize your macro for any other functions ?! Can you post your macro? :redface:

[jj2007]

I can post the macro, of course - don't know how useful it is. You will need Masm32 equivalents for Print and Locate.

Code Select Expand

gdi MACRO argc, args:VARARG
  invGdip @SubStr(<argc>, @InStr(1, <argc>, <Gdip>)), args
ENDM
invGdip macro gpCall, args:VARARG
Local tmp$
  ifidn <args>, <#>
call gpCall
  else
if @InStr(1, <args>, <esp>)
tmp$ CATSTR <## line >, %@Line, <: do not use esp as gdi+ argument ##>
% echo tmp$
if usedeb
  .err
endif
endif
push ecx
invoke gpCall, args
pop ecx
  endif
  ifndef invGdipOK
invGdipOK=0 ; >1 means show only failures
  endif
  if usedeb or invGdipOK
if invGdipOK eq 1
% echo gpCall args
endif
.if eax || invGdipOK eq 1
pushad
xchg eax, ecx
tmp$ CATSTR <Print "** line >, %@Line, <, &gpCall">
tmp$
PrintLine At(44, Locate(y)) Spc2$, gdiStatus$(ecx), Spc4$
popad
.endif
  endif
ENDM

Code Select Expand

gdiStatus$ macro inx
  push edi
  push ecx
  ExternDef GdiStatus$:BYTE
  mov edi, offset GdiStatus$
  xor edx, edx
  ifb <inx>
push GdiSI.jjLastError
  else
push inx
  endif
  mov al, 0
  or ecx, -1
  .Repeat
inc edx
.Break .if edx>stack
repne scasb
  .Until byte ptr [edi]==al
  pop eax
  xchg eax, edi
  .if Zero?
void Str$("Unknown error %i", edi)
  .endif
  pop ecx
  pop edi
  EXITM <eax>
ENDM

Code Select Expand

GdiStatus$ db "Ok", 0
db "GenericError", 0
db "InvalidParameter", 0
db "OutOfMemory", 0
db "ObjectBusy", 0
db "InsufficientBuffer", 0
db "NotImplemented", 0
db "Win32Error", 0
db "WrongState", 0
db "Aborted", 0
db "FileNotFound", 0
db "ValueOverflow", 0
db "AccessDenied", 0
db "UnknownImageFormat", 0
db "FontFamilyNotFound", 0
db "FontStyleNotFound", 0
db "NotTrueTypeFont", 0
db "UnsupportedGdiplusVersion", 0
db "GdiplusNotInitialized", 0
db "PropertyNotFound", 0
db "PropertyNotSupported", 0
db "ProfileNotFound", 0, 0

[LiaoMi]

Cool, many thanks!!! I will try to play with other functions!  :icon14: :icon14: :icon14: