Never ever override your instincts. If somebody doesn’t sound right on the phone (or in person), pass. Pass immediately and never look back. I’ve had those experiences a couple of times. Somebody wouldn’t sound right on the phone but, being the kind of person I am, I would be willing to give them the benefit of a doubt and give it a shot. I thought my hunch could be wrong  but regretted  such thinking every single time. I learned my lesson and it is very simple: trust your inner voice-if they don’t sound right, pass. It’s not worth it and it will turn into a nightmare. They may give you a malicious review or maybe even sue you. Those things haven’t happened to me but I believe it is because I precluded them by learning how to say no. 
If possible, it also helps to focus on business rather than individual clients; it’s easier to work with professionals.

If, on the other hand, you need that money so badly that you can’t refuse to provide service for them, make sure you charge sufficiently for your frustration.