There is bound to be a buildup of bacteria that thrive on the ammonia from waste in the tank. An excess of ammonia can be dangerous to your fish. There are strips that can test for the amount of ammonia in the tank. Anything above .5ppm is unsafe.Change part of the water in the tank, adding a water conditioner before adding fresh water. Testing and adding fresh water if there is a problem on a regular basis won't harm the good bacteria but will keep your fish healthy.