Welcome! If you absolutely must move him outside again, try getting him used to the cold in stages. Take him out for ten minutes the first day. Then take him out for 15 or 12. Keep increasing it until he is getting used to the cold. You can also give him things to keep him warm. A wooden box with plenty of hay will help, as will a hutch with solid back and sides.
As was mentioned, the allergies could be from the bedding. Shavings and straw will often cause allergies. Try a pelleted type of bedding. There is a pelleted pine bedding called Woody Pet (or you can use wood stove pellets), that would at least be better on allergies, though not perfect. There are also various pelleted newspaper litters, most made for cats or dogs, that would be even better for allergies.