Using optimized functions in Linux, uasm testcase

[mineiro]

Poor english language. I'm an eternal aprendice, not an expert. Contribute with critics and sugestion.

PART1 - INVESTIGATE
Suppose you have a strlen optimized function and like to test that inside other programs; your strlen function being called instead of default one inside C library (libc.so.6) by others programs.

Linux offers some tools to deal with binary, this is just an introduction about how to start.

Let's first explore executable/library/object files, I choose uasm because it's an assembler that I'm using.
NOTE: uasm executable file is present in same folder in all of these steps.
Download uasm at and decompress it to a folder/directory:
http://www.terraspace.co.uk/uasm.html

Let's try to execute uasm, open terminal, enter uasm folder and type:

mineiro@assembly:~/uasm$ ./uasm
bash: ./uasm: Negate permission

After download uasm can be necessary to give execute permissions to file:

mineiro@assembly:~/uasm$ chmod +x uasm

Let's try to execute uasm again:

mineiro@mineiropc:~/uasm/libuasm$ ./uasm
UASM v2.49, Jun 21 2019, Masm-compatible assembler.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.

usage: UASM [ options ] filelist [@env_var]
Run "Uasm -?" or "Uasm -h" for more info

Now uasm is acessible from this folder with execute permission. We can copy uasm to /usr/bin directory (with priviledged permissions like root), so it stays in global environment path, this way we can call uasm from any folder/directory. In other words, it's not necessay to insert a "./" prefix before executable file. Prefix "./" means to execute a program in current directory/folder.

nm command list symbols from object files (.o or executable files)

mineiro@assembly:~/uasm$ nm uasm
nm: uasm: (no symbol)

Hmm, uasm probably was compiled with strip (-s) option, so no results.
Let's check what library is necessary to uasm run, ldd - print shared library dependencies:

mineiro@assembly:~/uasm$ ldd uasm
        linux-vdso.so.1 =>  (0x00007ffdee523000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f80ad97c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f80ae030000)

First one I don't know what means, second is C library and third is loader.
Dynamic library in linux (.so) are equivalent to windows .dll. In linux libraries start with prefix "lib" name.
NOTE: perceive libc.so.6 is a symbolic link that points to real file located at /lib/x86_64-linux-gnu/libc.so.6
In linux you have symbolic links to files, so instead of typing a giant path and name you can create a simple symbolic file to that and refer to it.

We can disassembly uasm, redirect output to a file and check what functions are being used:

mineiro@assembly:~/uasm$ objdump -d -Mintel uasm > uasm.disasm

We can list functions being used by:

mineiro@assembly:~/uasm$ objdump -R uasm
...
00000000002a1e70 R_X86_64_JUMP_SLOT  strlen
...

mineiro@assembly:~/uasm$ objdump -T uasm
...
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strlen
...

So, we can trace uasm by using command:
strace - trace system calls and signals
ltrace - A library call tracer

mineiro@assembly:/bin$ ltrace uasm

You can perceive a lot of functions that uasm calls, one of that is strlen,tolower,... .

Uasm echo time spend in execution, one way to get an aproximation is using command time:

mineiro@assembly:~/uasm$ time uasm
real    0m0.003s
user    0m0.000s
sys     0m0.004s

Have other tools that is not included by default in some linux distros like a profiler called "valgrind", that checks for memory leaks, ... .


PART2 - CREATING A LIBRARY WITH STRLEN FUNCTION AND A PROGRAM THAT CALLS THAT FUNCTION IN LIBRARY
In Linux X86-64 we need follow calling convention and ABI. Search for "System V Application Binary Interface" in internet.
http://www.sco.com/developers/devspecs/gabi41.pdf
https://raw.githubusercontent.com/wiki/hjl-tools/x86-psABI/x86-64-psABI-1.0.pdf

A simplified example (without xmm use,...) of function/syscall argument is:
syscall rax,rdi,rsi,rdx,r10,r8,r9       ;system call
call        rdi,rsi,rdx,rcx,r8,r9       ;c library function call
Note that we can mix both ways in same program. Perceive R10 and RCX registers difference. Perceive that I don't show registers that need be preserved between calls and the ones that can be trashed. I don't show if function need more parameters/arguments. Read documentation.

Let's create a simple program that echo some text in screen and use strlen from libc.
usestrlen.uasm

Code Select Expand

;this file uses CRLF as new line, some linux editors by default use only LF so you need pay attention to this
;shell script files (.sh) uses LF, so pay attention.
.X64

STDIN                   EQU     0
STDOUT                  EQU     1
STDERR                  EQU     2

;unistd_64.h
sys_read equ 0
sys_write equ 1
sys_exit equ 60

;libbilioteca.so or libc
strlen proto :ptr

.data
hello db "hello world",10,0 ;Linux use LF(10), windows use CRLF(13,10) as new line.

.code
public _start
_start:

;invoke strlen, addr hello
    lea rdi,hello
    call strlen ;return in rax register sizeof string hello

;scall sys_write,STDOUT,@hello,rax
    mov rdx,rax
    lea rsi,hello
    mov rdi,STDOUT
    mov rax,sys_write ;write string hello to device screen
    syscall

;scall sys_exit,0
    mov rdi,0
    mov rax,sys_exit
    syscall

end _start


Let's assemble, create an object file.

mineiro@assembly:~/uasm$ ./uasm -elf64 usestrlen.uasm
UASM v2.49, Jun 21 2019, Masm-compatible assembler.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.

usestrlen.uasm: 40 lines, 2 passes, 4331 ms, 0 warnings, 0 errors

Now, let's link object file to create executable, note that "-lc" means use libc.

mineiro@assembly:~/uasm$ ld -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o usestrlen -lc usestrlen.o

And now let's run this:

mineiro@assembly:~/uasm$ ./usestrlen
hello world

One note is that loader path (/lib64/ld-linux-x86-64.so.2) can be different in your system, choose other and you can generate linux 32 bits elf executable.
Other note is that we can capture program exit code by appending " ; echo $?" to command line:

mineiro@assembly:~/uasm$ ./usestrlen ; echo $?
hello world
0      <--program exit code

So, let's create a simple dynamic library. I had choosen to download Agner Fog library (strlen64.asm) and adapt that. Our library will have only one function, nothing more.
https://www.agner.org/optimize/
Download and read that texts, here I'm using asmlib.zip
libstrlen.uasm

Code Select Expand

.code

align 16
public strlen
Rscopy equ rdi
strlen::
        mov      rax,  rdi             ; get pointer to string from rdi
        mov      ecx,  edi             ; copy pointer (lower 32 bits)

        ; rax = s, ecx = 32 bits of s
        pxor     xmm0, xmm0            ; set to zero
        and      ecx,  0FH             ; lower 4 bits indicate misalignment
        and      rax,  -10H            ; align pointer by 16
        movdqa   xmm1, [rax]           ; read from nearest preceding boundary
        pcmpeqb  xmm1, xmm0            ; compare 16 bytes with zero
        pmovmskb edx,  xmm1            ; get one bit for each byte result
        shr      edx,  cl              ; shift out false bits
        shl      edx,  cl              ; shift back again
        bsf      edx,  edx             ; find first 1-bit
        jnz      L2                    ; found

        ; Main loop, search 16 bytes at a time
L1:     add      rax,  10H             ; increment pointer by 16
        movdqa   xmm1, [rax]           ; read 16 bytes aligned
        pcmpeqb  xmm1, xmm0            ; compare 16 bytes with zero
        pmovmskb edx,  xmm1            ; get one bit for each byte result
        bsf      edx,  edx             ; find first 1-bit
        ; (moving the bsf out of the loop and using test here would be faster for long strings on old processors,
        ;  but we are assuming that most strings are short, and newer processors have higher priority)
        jz       L1                    ; loop if not found

L2:     ; Zero-byte found. Compute string length
        sub      rax,  Rscopy          ; subtract start address
        add      rax,  rdx             ; add byte index
        ret

end


Let's assemble:

mineiro@assembly:~/uasm$ ./uasm -elf64 libstrlen.uasm
UASM v2.49, Jun 21 2019, Masm-compatible assembler.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.

libstrlen.uasm: 37 lines, 2 passes, 4364 ms, 0 warnings, 0 errors
mineiro@assembly:~/uasm$ ld -shared /lib64/ld-linux-x86-64.so.2 -o libstrlen.so libstrlen.o
//mineiro@assembly:~/uasm$ ld --dynamic-linker -soname -shared /lib64/ld-linux-x86-64.so.2 -o libstrlen.so libstrlen.o

Now we need link our program (usestrlen) to use libstrlen.so instead of C (libc) library.

mineiro@assembly:~/uasm$ ld --dynamic-linker /lib64/ld-linux-x86-64.so.2 ./libstrlen.so usestrlen.o -o usestrlen

And execute:

mineiro@assembly:~/uasm$ ./usestrlen
hello world

Hmm, let's check what library usestrlen is using:

mineiro@assembly:~/uasm$ ldd usestrlen
        linux-vdso.so.1 =>  (0x00007ffc9fd01000)
        ./libstrlen.so (0x00007f31bd038000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f31bd239000)

So, we can copy libstrlen.so to some default path like /lib64 or /lib/x86_64-linux-gnu/
Note that program is working, but at next reboot it probably can't find libstrlen.so even being in same folder/directory.

;------Not used, just to demonstrate --------------
To create a static library (.a) instead of dynamic(.so), and recompile our program we should proceed this way:

mineiro@assembly:~/uasm$ ar r libstrlen.a libstrlen.o
ar: creating libstrlen.a

Not used here but if you have other modules (object files) just append that in command line above, like:

mineiro@assembly:~/uasm$ ar r libstrlen.a libstrlen.o othermodule.o module3.o

To list what modules are inside static library type:

mineiro@assembly:~/uasm$ ar -t libstrlen.a
libstrlen.o

We have an option to create an index in our static library by running command:

mineiro@assembly:~/uasm$ ranlib libstrlen.a

That can be show by command:

mineiro@assembly:~/uasm$ nm -s libstrlen.a

package index:
strlen in libstrlen.o

libstrlen.o:
0000000000000000 T strlen

So, let's link our program with this static library, run it and check dependency:

mineiro@assembly:~/uasm$ ld -o usestrlen usestrlen.o libstrlen.a
mineiro@assembly:~/uasm$ ./usestrlen
hello world
mineiro@assembly:~/uasm$ ldd usestrlen
        it's not a dynamic executable

This means that does not use external libraries. If you disassemble this file you will see that strlen function is inside executable file.
;---------------------------------------------

PART3 - LOADING OUR DYNAMIC LIBRARY WITH UASM.
abstract: uasm use dynamic C library; we created our own dynamic library that have strlen function.
Linux loader (ld.so) check to some environment variables and based in that we can insert our library to be loaded.
We can list environment variables by typing:

mineiro@assembly:~/uasm$ env

Let's set (export) some environment variables by typing:

mineiro@assembly:~/uasm$ export LD_PRELOAD=libstrlen.so
mineiro@assembly:~/uasm$ export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

Now, let's check again uasm with command:

mineiro@assembly:~/uasm$ ldd uasm
        linux-vdso.so.1 =>  (0x00007ffca4aeb000)
        libstrlen.so => ./libstrlen.so (0x00007f37709e3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f377061a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3770ecf000)

So, let's execute uasm but now using our strlen function inside libstrlen.so

mineiro@assembly:~/uasm$ ./uasm
UASM v2.49, Jun 21 2019, Masm-compatible assembler.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.

usage: UASM [ options ] filelist [@env_var]
Run "Uasm -?" or "Uasm -h" for more info

To reset environment variables we can type:

mineiro@assembly:~/uasm$ unset LD_PRELOAD
mineiro@assembly:~/uasm$ LD_LIBRARY_PATH=
mineiro@assembly:~/uasm$ ldd uasm
        linux-vdso.so.1 =>  (0x00007fffb8951000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8d335b7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8d33c6b000)