If someone you work with organized an office holiday dinner at a restaurant, and the high cost was the reason you weren't attending, would you tell them it's due to the cost?
I would absolutely tell why, as nicely as I could, that my government paycheck (which is probably not extravagant to begin with) will be going to the holiday celebrations with my loved ones, not to see the people I see every day at that job.