Is there an overdamped unstable system?

Not necessary that a overdamped systems will be BIBO stable. At some value of Gain, the ples might cross over to the right half plane. And the open loop stable system may become unstable.

Also note that there is the idea of Relative stability. A system may have Absolute stability but may have poor relative stability, which is usually true for overdamped systems. Which will have slow response.