Recent work studies the cognitive capabilities of language models through psychological tests designed for humans. While these studies are helpful for understanding the general capabilities of these models, there is no guarantee that a model possessing sufficient capabilities to pass those tests would actually use those capabilities in performing real-life tasks.In this work, we formulate task-oriented cognitive capabilities, which are human-like cognitive capabilities that language models lever…