Are you type 1 or 2?
I swim every day to avoid this problem, I know it's hard to predict how your body will respond once you get off your regular routine. I am a type 1 on an insulin pump, so I pretty much have two rates programmed into my pump; one is for regular swim days, the other is for other non-swim days. Non-swim says should still have some sort of structure... in fact it may be more important to have structure on these days! For me it means no eating carbs past a certain time of the afternoon, and not over-eating on carbs throughout the day. My insulin rate is higher on non-swim days, more especially in the night (but not significantly higher).
Hope you find the solution, whatever it may look like for you!