I've found this accepted Microsoft answer that lays out some conditions for syncing. I have a feeling that in your case it isn't transferring slow, but rather starting and stopping that is causing the delay.
Here are the conditions that must be met for a transfer:
Your phone and PC are both connected to your home Wi-Fi network. Note that wireless sync is not supported on workplace Wi-Fi networks.
Your phone battery has at least a medium-level charge and your phone is connected to an AC power adapter (that is, your phone must be connected to the wall charger and not connected to your computer).
You're not interacting with your phone. For example, you're not touching things on the screen, the backlight is off, and music and videos aren't playing (it's OK if a song or video is paused).
I hope this helps, good luck.