Ok - so this got me intrigued about what is possible. There's surprisingly little available, and whatever is available is really really old. What little I could find is related to Google TV pairing protocol (for connection) and Anymote (for sending commands). Anymote is also an app available on the play store and it is able to control the TV. Some code for a 'Google TV remote' app that might give inspiration - https://github.com/NineWorlds/google-tv-remote (though it's 8 years old).

The reason why I think this may work inspite of being really old is, if you go to the open source licences page of the Android TV remote control app (also available on the play store, made by Google), then you see a reference to google-tv-pairing-protocol (which is also used in the remote control app whose source code I linked above).

I'm honestly a bit surprised and disappointed how little is developed and documented for this platform. I'm hoping I'm missing something.

Home Assistant has a component that integrates with Android TV https://www.home-assistant.io/integrations/androidtv/ .

It uses adb to control the device, so even if you aren't interested in Home Assistant it might be a good place to start looking.