This sounds like a good overall strategy, but I would suggest the option be included of not storing the private key on the webserver at all. In that cage, it could be stored on the site admin's local PC and copied and pasted into a textbox on the page for viewing order details.

Or perhaps, having the order page show an encrypted credit card number that the user has to download to decrypt using local decryption software.