While I was working on a big project full of macro tricks and wizardry, I stumbled upon a bug in which a macro was not expanding properly. The resulting output was "
EXPAND was defined as "
#define EXPAND(X) X", so clearly the output should have been "
"No problem", I thought to myself. "It's probably some silly mistake, there are some nasty macros here, after all, plenty of places to go wrong". As I thought that, I isolated the misbehaving macros into their own project, about 200 lines, and started working on a MWE to pinpoint the problem. 200 lines became 150, which in turn became 100, then 20, 10... To my absolute shock, this was my final MWE:
#define EXPAND(X) X #define PARENTHESIS() () #define TEST() EXPAND(0) EXPAND(TEST PARENTHESIS()) // EXPAND(0)
To add insult to injury, almost any modification to the macros will make them work correctly:
#define EXPAND(X) X #define PARENTHESIS() () #define TEST() EXPAND(0) // Manually replaced PARENTHESIS() EXPAND(TEST ()) // 0
#define EXPAND(X) X #define PARENTHESIS() () #define TEST() EXPAND(0) // Manually replaced TEST() EXPAND(EXPAND(0)) // 0
// Set EXPAND to 0 instead of X #define EXPAND(X) 0 #define PARENTHESIS() () #define TEST() EXPAND(0) EXPAND(TEST PARENTHESIS()) // 0
But most importantly, and most oddly, the code below fails in the exact same way:
#define EXPAND(X) X #define PARENTHESIS() () #define TEST() EXPAND(0) EXPAND(EXPAND(EXPAND(EXPAND(TEST PARENTHESIS())))) // EXPAND(0)
This means the preprocessor is perfectly capable of expanding
EXPAND, but for some reason, it absolutely refuses to expand it again in the last step.
Now, how I'm going to solve this problem in my actual program is neither here nor there. Although a solution would be nice (i.e. a way to expand the token
EXPAND(TEST PARENTHESIS()) to
0), the thing I'm most interested in is: why? Why did the C preprocessor come to the conclusion that "
EXPAND(0)" was the correct expansion in the first case, but not in the other ones?
Although it's easy to find resources on what the C preprocessor does (and some magic that you can do with it), I've yet to find one that explains how it does it, and I want to take this opportunity to understand better how the preprocessor does its job and what rules it uses when expanding macros.
So in light of that: What is the reasoning behind the preprocessor's decision to expand the final macro to "
EXPAND(0)" instead of "
Edit: After reading Chris Dodd's very detailed, logical and well-put answer, I did what anybody would do in the same situation... try to come up with a counterexample :)
What I concocted was this different 4-liner:
#define EXPAND(X) X #define GLUE(X,Y) X Y #define MACRO() GLUE(A,B) EXPAND(GLUE(MACRO, ())) // GLUE(A,B)
Now, knowing the fact that the C preprocessor is not Turing complete, there is no way the above will ever expand to
A B. If that were the case,
GLUE would expand
MACRO would expand
GLUE. That would lead to the possibility of unlimited recursion, probably implying Turing Completeness for the Cpp. So sadly for the preprocessor wizards out there, the above macro not expanding is a guarantee.
It failing is not really the problem, the real problem is: Where? Where did the preprocessor decide to stop the expansion?
Analyzing the steps:
EXPANDand scans in argument list
GLUE(MACRO, ())as a macro:
MACRO ()for macros, finding
GLUE(A,B)for macros, finding
GLUE. It is suppressed, however, so it leaves as is.
Xafter step 2 is
GLUE(A,B)(notice that since we are not in step 4 of
GLUE, in theory, it is not suppressed anymore)
GLUE(A,B)for more macros, finding
Bfor the arguments (oh no)
A Bfor macros, but finds nothing
Which would be our dream. Sadly, the macro expands to
So our question is: Why?
For the purposes of this situation, there are three relevant steps in macro replacement:
TESTis not followed by parentheses, so it is not interpreted as a macro invocation.
PARENTHESIS()is a macro invocation, so the three steps are performed: The arguments are empty, so there is no processing for them. Then
PARENTHESIS()is replaced by
()is rescanned and no macros are found.
EXPAND(TEST ()). (
TEST ()is not rescanned because it was not the result of any macro replacement.)
EXPAND(TEST ())is replaced with
TEST ()is rescanned while suppressing
TEST ()is replaced by
EXPAND(0)is rescanned, but
TESTare empty, so there is no processing.
TEST ()is replaced by
EXPAND(0)is replaced by
EXPAND(TEST ())has become
EXPAND(0)is replaced by
0is rescanned for further macros, but there are none.
The other examples in the question follow similarly. It comes down to:
TEST PARENTHESIS(), the lack of parentheses after
TESTresults in it not being expanded while processing arguments to an enclosing macro invocation.
PARENTHESISis expanded, but this is after
TESTwas scanned, and it is not rescanned during processing of the argument.
TESTis rescanned and is replaced then, but, at this time, the enclosing macro’s name is suppressed.