You're not the only one experiencing this:
I'm attempting to switch to the Lumia 820 from previous Lumia phones
-- the 920 and the 720. Sadly, I'm seeing an issue that is also present on the Lumia 720, i.e.: with an SD card inserted, the camera
takes longer to snap the first photo. The proceeding photos are
typically snappy, but if you wait too long it seems to lag again. With
the SD card removed, all photos snap quickly and consistently.
Exact issue is described here: micro SD card mounting delay my Lumia
820 to take picture - Microsoft Community
I've tried several SD cards -- Sandisk 32GB Class 10 (current),
Kingston 64GB Class 10, and a Toshiba 4GB Class 4 -- all of them with
the same results. The phone is also noticeably more laggy in other
areas when the SD card is installed: Settings, Xbox Music, etc..
This is probably the reason why Apple device's don't have an sdcard: a) it introduces another failure and support point and b) It slows down the device because most operations want to check if the card is there first.
The Settings screen loads slower because it seems to check how much free space for the summary.
The home screen loads slower because it might have to fetch icons for apps or pictures for thumbnails from the gallery.
So in closing, I'm not sure there is a solution for this other than to not use an SDCard, which limits your storage size, effectively rendering your phone to just be a communications device and not a multimedia entertainment device.
I don't think any manufacturer can really work around this, unless they carefully design their system to not hit the disk unless specifically requested.
With the Lumia Black update (2013/2014), I notice that some of these slowness issues are performing better in my mind. It seems Microsoft have improved the SD card handling and made some speed fixes.