To clarify on the question, I have the Shield TV and I bought it mainly for responsiveness and the addition of gaming capabilities.
I have a 4K Sony TV that came with android TV function and it’s just horrible. It’s underpowered to the point even the UI lags. It frustrated me to the point that I just exploded and came home within the hour with the Shield TV in hand.
The Shield TV is great. Very responsive with some gaming capabilities. Although I find it great, I don’t use its gaming capabilities as often as I thought I would.
I’m basically wondering if I can return the Shield TV, go the $50 Google TV, and still have the responsiveness I want.
Just keep the Shield, it's the best Android TV device that I'm aware of. There's no reason to downgrade your experience to save a few bucks.