Yeah, there are failsafes in the code that prevent you from accidentally deleting a valid order. This is based on order status and payment history. If there are any logged payments for an order, you'd need to delete them first and move the order back to Pending in order to delete it. You can use the icon from the order admin screen or the delete button on the edit form. My plans include making a way for store owners to turn off these failsafes, but I haven't gotten to that yet.