The problem comes from people that are not breaking and entering to be there. A person's privacy is a sticky subject, hence the recommendation to find out what is legal and illegal in your area. Just because your boss is using common sense doesn't mean it would be legal to do what he wants to in your area.
Again, just recording video is generally not a problem, especially if you notify people as they come on premise through signs. It is the audio recordings that can get you in trouble depending on where you are located.
If the systems were separate (ie a CCTV system that recorded video only and an intercom system that didn't record audio), I doubt you would have any problems. Then if something was going on, a person could use the two way audio to speak into the situation even if they are off site. There are plenty of alarm systems that do this with the audio portion. For example, when there is an alarm a person can listen in and speak to the premise through the alarm system.