Can I only take panoramic photos from left to right?

No. There are four supported directions: left-to-right, right-to-left, top-to-bottom and bottom-to-top (tap on Q button in Panoramic Mode to toggle between these directions). An arrow will be displayed to guide you in the right direction.