Well all I had to do was add a forward slash to the url like this

<a href="/<?php print $node->links['comment_add']['href']; ?>#comment-form">Add Comment</a>

Now it adds the proper url to the form anchor. I'm sure there is a cleaner way of doing this but I know nothing about programming so...