We've been able to replicate this, so will mention it to the dev team and look to get it added to the backlog.
This seems to be quite a specific case-use scenario that depends on having an empty queue, and beginning to build the queue by using 'Add To Queue' rather than starting by playing a track.
In the meantime, here is potential workaround:
With SmartQ turned off, when starting to build your queue do not use 'Add to Queue' on your very first track. Instead, use the 3 dots/more options on the desired song to 'Play Track'. This will start it playing - feel free to immediately Pause it if you wish.
Subsequent tracks that then get Added to Queue after this should not disrupt the playback or cause any unexpected skipping to next tracks.