A Relationship between Function Pointer and Function Calling Convention in "C"

C - Function pointer & Calling Convention - Relationship:
    There is a unique relationship between function pointer and function calling convention in C. The function calling convention is one of the parameters that define the signature of a function. Each function pointer can be used to manipulate the functions with a particular signature. So, a function with different calling convention cannot be manipulated by the same function pointer. 

What is Function Calling Convention in "C":
     
    When ever a function is called in C, a stack frame is created in the stack area for that function. Stack frame stores several information relevant to that function. The arguments passed to the function, local variables, return address, etc will be stored in that stack frame. When the execution of the function completes, the stack frame will be destroyed.
    The C function calling convention tells the compiler things such as

• The order in which the function arguments are pushed onto the stack.
• Stack Cleanup Responsibility: When the function execution completed, whether the caller or called function will take care of destroying the stack

    The calling convention belongs to a  function's signature. To know about what function's signature implies, click here. Some of the calling conventions used in C compilers are __stdcall, __pascal, __cdecl and __fastcall. Every compilers follow their own way to specify the calling convention during the function definitions/declarations. Because of this, it may be possible to get linking errors such as "unresolved external" if we try to link object files compiled with different compilers.

    For example, the Borland and Microsoft compilers mandates the specification of calling convention type between function return type and function name.

    //Borland and Mircrosoft calling convention specification
   <FnReturnType> <calling_convention_type> FnName(argument)

   Example: void __cdecl TestFunction(char *str);

    For the GNU GCC, the __attribute__ keyword is used to specify the function calling convention.
    <FnReturnType> __attribute__((cdecl)) FnName (arguments)

    Example: void __attribute__((cdecl)) TestFunction (char *str) ;

Function Pointer & Calling Convention Relationship:

    A function pointer always points to a function with specific signature. So, we cannot use the same function pointer to point functions with two different signature. As the calling convention of a function is one of the signature parameters, we cannot use the same function pointer to manipulate a function with two different calling convention.

Illustrative Example:
     Consider the below C program to understand the concepts introduced above.

#include <stdio.h>
int add (int,int); /* Function with default calling convention */
int sub (int,int); /* Function with default calling convention */
int __attribute__((fastcall)) sub_new (int,int) ; /* Function with __fastcall type calling convention */
int main()
{
    int var1 = 10, var2 = 20, rslt = 0;

   
    int (*FunPtr) (int,int) = NULL; /* A function pointer which accepts a function whose return
                                       type is integer and accepts two integer arguments and the
                                       function with default calling convention*/

    int __attribute__((fastcall)) (*NewFunPtr) (int,int) = NULL;  /* A function pointer with non-
                                                                                                          default calling convention */

    FunPtr = &add;

    rslt = FunPtr (var1,var2); /* Calling the function add */
    printf ("Addition Result:%d\r\n",rslt);

    FunPtr = sub;

    rslt = FunPtr (var1,var2); /* Calling the function sub */
    printf ("Subtraction Result:%d\r\n",rslt);

    FunPtr = sub_new; /* We are assigning a function with different calling convention */

    rslt = FunPtr (var1,var2); /* Calling the function sub_new */
    printf ("Subtraction with __fastcall calling type -->Result:%d\r\n",rslt);

    NewFunPtr = sub_new; /* We are assigning a function with proper calling convention */

    rslt = NewFunPtr (var1,var2); /* Calling the function sub_new */
    printf ("Subtraction with __fastcall calling type and __fastcall Fn.Ptr --> Result:%d\r\n",rslt);

    return 0;
}

int add (int val1, int val2)
{
    return (val1+val2);
}

int sub (int val1, int val2)
{
    return (val1-val2);
}

int __attribute__((fastcall)) sub_new (int val1, int val2)
{
    return (val1-val2);
}

Compilation Warning:
    When we compile this using gcc, it throws warning for possible incompatible pointer type assignment at the line "FunPtr = sub_new;"

root@ubuntu:~/Programs# gcc -Wall FnPtr2.c
FnPtr2.c: In function ‘main’:
FnPtr2.c:25:12: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
     FunPtr = sub_new; /* We are assigning a function with different calling convention */
            ^
root@ubuntu:~/Programs#

Output:

root@ubuntu:~/Programs# ./a.out
Addition Result:30
Subtraction Result:-10
Subtraction with __fastcall calling type -->Result:-938653831     <<< junk value is returned when we used function pointer with different calling convention.
Subtraction with __fastcall calling type and __fastcall Fn.Ptr --> Result:-10   <<< Result is proper when the calling convention matches.
root@ubuntu:~/Programs#


References:

 https://www.tenouk.com/Bufferoverflowc/bufferoverflowvulexploitdemo31.html