After removing or adding some items from the Panel in XFCE, you may notice that all the items have become left-aligned and collected on the left side of the Panel.
XFCE distributes the items on the panel in a left-to-right manner. To keep some items on the right side, you need to add an expandable separator between them. This is listed as Separator in the Panel Preferences. To make it expandable, click on the Separator item and then click the gear icon to edit its properties. Here you will find an option to make it expandable. Once you add an expandable separator, the items will be aligned to the left side of the Panel.
Tried with: XFCE 4.12 and Ubuntu 15.04