You could try telling people that the chair is where your imaginary friend sits - that should scare them away

.
The only thing I can suggest is that when people sit down, you ask whether you can help them (with the biggest, sweetest smile you can manage) and if they say no, then just reply "I'm sorry, but I have something important to finish right now ..." Nine times out of ten, they will go away, but you will still be friends.
The thing is that such a chair will invite people to sit down. I have a visitor chair in my office and people DO visit, which is fine because I'm happy to see my colleagues or directors of my organisation. They can distract from my work, but networking is work too.
Raindance