Food rewards are a great form of positive reinforcement for tasks well done. Typically, the more the horse likes or wants a reward, the harder they will work for it. Likewise, if he is full after a meal or doesn’t really fancy the treat, he may be less motivated by a food reward. Ideally, treats are novel enough for the horse that he recognizes them as a unique treat for training, but they also shouldn’t wreak havoc with the rest of the diet.
I think carrots or apples (the classics) are the easiest and most convenient reward. They can be cut up and put into a bag or pocket to carry around you as you work with your horse. Both are also high in water content, so the sugars within them are relatively low and safe for most horses (unless a horse is very sensitive to sugar in its diet). Some owners worry about toxicity from beta-carotene in carrots, but because beta-carotene is actually water soluble (unlike vitamin A, which beta-carotene is converted to if it is needed) there shouldn’t be any problems.
Hay cubes are also convenient and safe, and provide added fibre to the horse’s diet. If you feed a lot of them, however, the amount fed can add up and you might need to adjust your horse’s daily ration accordingly (particularly if your horse needs to count its calories!).
Many commercial treats are good too, but if they are dense, have oats, corn or molasses in them, the calories and sugars can add up quickly. I would use those in moderation for the very special rewards.