Orange Squeeze always leaves a notification up when I leave the app, can I disable this?

Yes, starting in version 2.3.1 there is a preference to disable the now playing notification. Even when enabled, the notification should only remain as long as music is playing.