Physical unclonable function (PUF) has been widely applied as a security primitive in multiple authentication schemes. It is usually adopted in the chip design and used to generate a unique key to authenticate one with key to operate the design normally. PUF response hence need be retrieved in a secure way rather than be accessible to anyone. However, this issue is neglected in some existing work. To highlight the importance of this issue, we first identify the scenarios where secure retrieval of PUF response is necessary. We then propose the principle to retrieve the PUF-based key in a secure way and discuss the evaluation criteria. We survey, classify and evaluate the existing work on this issue and analyze the advantages and disadvantages of each scheme. Suggestions are finally given for the designer how to select appropriate retrieval scheme in a specific hardware design.