I don't know the specifics of your installation, but if at all possible, try to slope the majority of the line towards the outside vent. That allows any condensation that builds up to drain off outside the building and not back into your fan.
As for the insulation, I just put an extra layer of fiberglass batts over my vent lines in the attic space (ours vent out the gable end walls). They're probably better insulated than the rest of our house.
You could also wrap the pipe with fiberglass batts and use string, duct tape, plastic zip ties, or something similar to hold it all in place.
Alternatively, use long strips of insulation, and start wrapping at one end, overlapping the previous wrap by a 1/2 width or so with each turn. This would give you about twice the insulating value and no gaps in the insulation.
Anthony