Nobody is going to try to stop you from going to a residency interview. Applying to residency, going to interviews, and locking down a residency spot is the whole point of medical school.
I have an interview this Friday (and possibly another on Monday...it's in the works), and I just let my next rotation (surgery) know I won't be in until Tuesday. They were fine with it...just made sure I understood I would have to arrange for lessons on scrubbing in once I get back since I'll be missing orientation. Just do what you have to do...as fast as you can...and get back to your rotation ASAP.