I have a Chromecast with Google TV. From the app browser I installed an apk downloader and sideloaded Chrome. In chrome, I turned on account sync, and browsed straight to my G-Mail. No password prompt.
I assume I could do this at anyone's home.
EDIT: TL/DR your chromecast is effectively an always-unlocked cell phone, signed in to your Google account.
You did turn on account sync, what did u expect?