Most object detection methods require abundant labeled training samples per category. However, we often receive a continuous temporal stream of visual data with many unknown categories in real-world applications. For example, a robotic learning system needs to deal with new objects observed in the environments but cannot store historical samples or re-train the model due to storage and computation limitations. Existing methods usually suffer from either catastrophic forgetting of the old categories or failures in the unknown novel categories. To alleviate the above issue, we propose to study a new task, continual open-set object detection. "Continual" means we observe training samples phase-by-phase. We need to train an objection detection model for all observed categories without the previous data. "Open-set" means we are required to deal with unseen novel categories from the stream effectively. We need to not only accurately classify the seen categories but also discover the new categories automatically. Our target is to develop object detection algorithms that work in both continual and open-set settings.