Because you happen to be assigning your tokens doesn’t mean that validators can take control of your coins. They don’t keep or very own your assets. At any given time a delegator can un-delegate their stake and delegate with other validator. Ethereum’s mining works a bit in another way https://financefeeds.com/wyoming-set-to-introduce-state-issued-copyright-stablecoin/