That's perfectly normal. He's trying to adjust to his new surroundings, and he's probably exploring the cage while you all are asleep. As long as you bond with him efficiently, he'll tweak his sleeping schedule to wake up earlier so you guys can hold him and play with him. My hamsters would consecutively wake up between 5 - 6 pm, knowing that as soon as I saw them awake, I'd pick them up.
You really do need to get him used to your scent, though, so he doesn't get accustomed to you not having your hand in his cage. What kind of cage is he in? My hamsters always freaked out in wire cages, they preferred the security of an aquarium, or any type of cage that didn't expose them like wire cages do.
Try offering him a piece of food, and if he doesn't take it, put it near him. Try doing that several times a day.
I earned my hamster's trust by giving them a situation in which to decide to trust me, or not. Whenever I get a hamster, I leave the box they came in in their cage with them. They'll often use it to nest because it's a dark area that already has their scent. Generally some time after they slowly edge out and start exploring, they'll get on top of the box and seem unsure of how to get down, that's when I put my hand, palm up, right next to the box. And they can decide whether to use my hand to get down(if they crawl onto it, I'll slowly decline my hand to the bottom of the cage, and wait until they walk off of it on their own before removing it), or they'll hop down on their own.
I don't like the method of just leaving your hand in there without a purpose when they're still trying to get used to everything, because generally, they'll just avoid it anyway, and it probably makes them feel like it's an invasion of their territory. Which is why, when introducing them to my scent, I first keep their box in the cage so they already have a dark area with their scent in a new, foreign cage, and whenever I put my hand in there, it's never in there for too long, and it's either to help them down from on top of the box, or to offer food. At first, when they take the piece of food from me(pinched between my fingertips), they'll snag it and run away to their safe place. But eventually they'll nibble at it and continue to let me hold it for them, or gently take it and sit right there eating it,
When they get to that point, it's a safe bet that you can leave your hand in there at prolonged periods of time without fear of being bit, just make sure every time your hand is in there, it's doing something useful, so he doesn't associate your hand exclusively with food. Try rustling the bedding up some away from him, so he gets used to disturbances, and so your scent gets on his bedding.