Then you've got a big problem that your neighbor thinks he can access your property without permission wherever and whenever he wants. Make a police report for documentation. Perhaps have him tresspassed. He brought a gun because he thought he might need to shoot you.