Has your cat had much socialization? If not, it takes a bit of time and patience to gain his trust. Some cats are intimidated by a human's size so crouch down to pet him. Cats are predators, so using toys, especially the fishing pole types, can encourage your cat to come out of his shell. Don't make any loud noises around the cat that may startle him. Your cat can sense your feelings so keep calm around him. Ask visitors to behave calming near your cat too. You can also try Bach's Rescue Remedy for Pets to calm him down, just follow the instructions for cats. Eventually, with patience your cat will lose his fear.