It all depends if you trust the client, whether you feel comfortable working in such an environment and are you willing to walk away from the client. If there's no trust nor feeling at ease working there and you're willing to walk, do so. Only you can answer these questions for yourself. But be open to talk about your working relationship. You might gain more respect from them by doing so.