Does removing users remove their hours also?
No, removing a user from your workspace does not delete their hours.
You won't be able to filter them out one-by-one, but the times they have tracked so far will remain accessible to you in your workspace when you keep the user filter as 'Everybody'.