What you have is what we call "Arm Pump". It is caused by over gripping the hand controlls. To alleviate the problems:
1. Relax. This is the number one culpret for arm pump.
2. Nice set handle bars. Prefferably Fasst Flexx. They have rubber bumpers that soak up much of the bumps.
3. Good steering stabilzer. Presicion is about the best you can get on the market. This will greatly reduce fatigue.
4. Upgrade your suspention or at least get them revalved and set up for your style of riding.
5. Change the angle of your thumb throttle. Usually they are set at a general angle at the dealership. They mechanic sets them for his hands, which aren't yours.
These are some of the more commonly means of reducing fatigue. The most important though is to relax. Move with the bike and try not to fight it.