I want to create a linux shell with overridden malloc() i.e. malloc_hook(). My terminal should take an .exe file as input and run the program as a thread. Whenever there's a malloc() call in the program, terminal's malloc_hook() should be called eventually. How can I achieve this goal. I have gone through a tutorial to write my own Linux terminal link : https://abhijangda.wordpress.com/2013/04/14/creating-a-simple-linux-terminal-in-c/ but that wasn't just enough.
Using malloc_hook does not have much to do with the Linux terminal. You can look at the man page for malloc_hook to try and understand how it works. but it seems to have been deprecated in recent versions of the glibc:
http://man7.org/linux/man-pages/man3/malloc_hook.3.html
You can also read the manual page for mallopt for other ways to tweak the allocation system:
http://man7.org/linux/man-pages/man3/mallopt.3.html
If the program is yours, use malloc_hook as documented above.
If you cannot recompile the program, you may achieve your goal by making a library that replaces all of the malloc family of functions and dynamically link the program with this library before the C library, using LD_PRELOAD as documented here:
http://man7.org/linux/man-pages/man8/ld.so.8.html
And as discussed in this question: What is the LD_PRELOAD trick?