Is he eating, drinking, and going to the bathroom?
First of all, 6 weeks is too young for him to be away from his mother. 8 weeks is the normal age for baby rabbits to be taken from their moms. In some US states, it's even illegal now to take them younger than this. I'd keep a very close eye on him, and take him to the vet if you notice anything wrong - especially if he stops eating or using the bathroom.
He probably needs more time to settle down in his new environment. 5 days is not a long time when you consider that the little guys entire world has just changed. Some rabbits tend to be shy/timid in the first place, and then the stress of going to a new home just adds to that. You're going to have to give him time to settle down.
You will have plenty of time to develop a bond with him later on, so don't worry about missing a few days or so. Rabbits can be sensitive creatures and really bonding
with them can take a lot of time and patience anyways. It's not always something that will happen quickly. Right now I think you really just have to give him time to get used to his new surroundings.
I'd leave him in his cage for a few days. Let him get used to his cage and start to understand that it's his, and is a place he can feel comfortable in. Once he feels more secure in his cage then you can start getting him used to exploring outside of his cage and work on bonding with him more.