Whether or not you were rude in this situation depends on your tone and manner of refusal. It's essential to consider the context and your reasons for not walking with them.
If you politely declined their request and gave a valid reason, such as having time constraints, personal errands, or simply wanting some alone time to mentally prepare for work, it's unlikely that you were rude. People have different schedules and personal boundaries, and it's essential to respect that.
On the other hand, if you were dismissive, abrupt, or insensitive in your response, it might be perceived as rude. How we communicate our boundaries matters. If you feel that you might have come across as rude unintentionally, you could consider reaching out to your coworker and explaining your perspective in a kind and understanding manner.
Ultimately, communication is crucial in such situations, and being respectful and considerate of others' feelings is always a good practice.