I guess if you view faculty prestige as more of an aggregate of all faculty members, and UCSC was pretty small, then you could have a high impact factor (since that is measured per-paper-published) without having a necessarily high faculty prestige (which might be more analogous to most citations in general).

I mean, the way it appears to be measured, if you took the exact same faculty you guys have now, and hired 5 more profs with slightly lower average-citations-per-paper, then your "impact factor" would actually go down. However, it could certainly be the case that your department would be considered "more prestigious" or "better in general" if it was your current faculty plus five more pretty good profs, right? Maybe something like that helps explain what is going on.