Eating with co-workers helps one know and work with them better.
Refusal may be interpreted as rejection or lack of interest.
Sharing meals is a natural way to get to know one other.