Overt behavior doesn't demonstrate understanding
The Chinese Room argument shows that a system can engage in Chinese-speaking behavior yet fail to understand Chinese. The Chinese Room is a system that could past the Turing Test without thinking in Chinese.
John Searle, 1980b.