The main problem that has been brought up by a myriad of commentators is that Google Duplex seems inherently built to deceive. The whole premise of the technology is to fool someone on the other end of the phone call into thinking that Duplex is a human assistant making a booking.
That can be seen as a mild enough task with no dangers, but now imagine the very same technology being used against you. Along with Duplex, Google also showed how they could use machine learning to learn a voice in much less time compared to before.
What if something like Duplex with your voice called your employer and said some nasty things?
What if you got a call from someone you trusted and inadvertently gave out some personal information, only to find that the person on the other end of the phone was a bot?
Not only will this technology be able to deceive, but as a piece of software, it will be able to do it at scale. To millions of people.