qt: make some popups non-modal and fix popups do not close on press outside with `CompositorX11`

I don't see a reason why tracks or tool button popups should be modal. It seems that they were made modal because of a Qt 5 bug, this is no longer relevant to us.

Request review @chub.