There are conservative and hybrid garbage collectors.
The conservative ones just look for bit sequences looking like pointers. The hybrid ones require the programmer to specify some layout information explicitly in the code (but they still traverse the call stack conservatively when they look for pointers). Garbage collectors may cause memory leaks when a bit pattern looking like a pointer is misinterpreted as a proof that the block pointed by this "pointer" is still in use. And some illegal programs may "confuse" garbage collectors by keeping pointers outside of allocated blocks.