Am I being racist in this situation?

No it's not racist but it IS unfair. It's unfair to your kids and unfair to their friends. My best friend in elementary was russian and her parents didn't speak a word of english. I slept over her house tons of times and visa versa. There was never a problem. Many people speak Spanish and you could probably find someone to interpret if you needed to.