If you don’t want the site accessible, don’t host it. Play with it in a sandbox or something.
If you want it available on the net, but only to you or people you authorize, put in some authentication to access it.
Otherwise, what are you planning on doing with the information you get? Knock on their door and ask them to stop visiting your website? Hardly seems reasonable. You put a website on the net, expect people from the net to visit the website.
IP has very little useful information, can be easily changed, and easy to spoof. You’re not going to get far with just an IP – whatever your plan is.