Mosh is a replacement for SSH that uses UDP streams. However, it still connects over SSH to establish the connection.
Here is how to connect to a custom SSH port when using mosh by setting the flags used to establish the connection.
mosh firstname.lastname@example.org --ssh="ssh -p 2222"
You can simply start mosh from the command line without an IP to see which other flags are available.