It won't knock you out of ketosis, but it will impede your weight loss. In order to lose weight you have to eat at a calorie deficit. So if you are just doing keto for health, don't worry about calories. If you want to lose weight, worry about calories.