Give it time.
My GP,Stuart,was a rescue and when I first got him in June (I was only fostering then...ended up adopting!

) he was extremely shy.He ran away from me,sat in corners and freaked out when I tried to handle him.
I took my time with him and been slow.I sat in his cage (it is big enough) and offered carrots.At first he wouldn't come over,but after awhile he couldn't resist.He soon realized that I bring good things.
He has come ALONG way since I got him in June.He now runs up and greets me,lets me pick him up and handle him,lets me trim his nails and is constantly talking to me.It took patience,time,love,food and talking softly.
Stuart is about 3 or 4 yrs old and came from a bad home,yet he quickly learned to trust me.Since your little one is a baby,I am sure it won't take long for him to warm up to you as well.
Good luck.