SDBench is an open-source benchmark suite proposed to address the high variance in error rates of state-of-the-art speaker separation systems across multiple datasets representing diverse use cases and domains. It integrates 13 diverse datasets and provides tools for consistent and granular speaker separation performance analysis, enabling reproducible evaluations and easy integration of new systems. To demonstrate the effectiveness of SDBench, we build SpeakerKit, a system focused on inference efficiency based on Pyannote v3. We evaluate SpeakerKit's performance using SDBench and show that it is 9.6x faster than Pyannote v3 while achieving a similar error rate. We also benchmark six state-of-the-art systems, including Deepgram, AWS Transcribe, and the Pyannote AI API, to uncover the critical tradeoff between accuracy and speed.