Even when you think your private conversation won't be heard by others, it almost always is. The other Castaways use the overheard information to their advantage amongst their own Tribal alliance.