The boundaries that you set serve as self-love and help you keep your mental balance. You deserve peace and not guilt. Guilt is a means to force control over the choices one makes- - do not allow it.