My application uses static libs compiled with gcc: Boost( C++11 lambdas (with boost bind and boost function)) Bullet; system shared libraries such as SDL, and one shared library compiled with clang. Is it possible that such zoo would mess up AdressSanitizer?
Yes if they're not instrumented:
https://code.google.com/p/address-sanitizer/wiki/AddressSanitizer
In order to use AddressSanitizer you will need to compile and link your program using clang with the -fsanitize=address switch.
The ASan runtime library must be present in your program, so your main executable needs to be linked with the -fsanitize=address flag. Linking non-instrumented and instrumented libraries together may work, unless instrumented code is executed before the runtime library is initialized (I think it's impossible on Linux right now). Note that AddressSanitizer won't be able to find addressability issues in the code that wasn't instrumented by Clang.
Clang sanitizers can be used as below
Download clang toolchain from here http://releases.llvm.org/download.html
$ wget http://releases.llvm.org/9.0.0/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz
$ tar -Jxvf clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz
$ CLANG=$PWD/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-16.04
Sample program
$ cat hello.c
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
int *a = (int *) malloc(sizeof(int)*2);
int n = atoi(argv[1]);
a[n] = 10;
return 0;
}
Compile the program with address sanitizer
$ $CLANG/bin/clang -O0 -g -fno-omit-frame-pointer -fsanitize=address -o hello hello.c
Run
$ ASAN_SYMBOLIZER_PATH=$CLANG/bin/llvm-symbolizer ./hello 12
=================================================================
==48489==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x602000000040 at pc 0x0000004c2981 bp 0x7ffe3f888c30 sp 0x7ffe3f888c28
WRITE of size 4 at 0x602000000040 thread T0
#0 0x4c2980 in main /b/syrajendra/clang-libs/hello.c:8:8
#1 0x7f0349c3a82f in __libc_start_main /build/glibc-Cl5G7W/glibc-
2.23/csu/../csu/libc-start.c:291
#2 0x41b2f8 in _start (/b/syrajendra/clang-libs/hello+0x41b2f8)
Address 0x602000000040 is a wild pointer.
SUMMARY: AddressSanitizer: heap-buffer-overflow /b/syrajendra/clang-
libs/hello.c:8:8 in main
Shadow bytes around the buggy address:
0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa fa 00 fa fa fa fa fa[fa]fa fa fa fa fa fa fa
0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==48489==ABORTING