Have camera, will shoot. There are very thin lines between what is acceptable to film and what is not. There are a number of things to consider.

First, whether the location is public or private. A public space like a park or a main thoroughfare is a free-for-all space. However, when shooting in private areas like residential areas and offices, there must be an understanding between you and the owners/administrators of the building of what the video will be used for.

Second, who are the subjects. Bystanders and passersby do not need to be asked for permission unless they are your primary subjects and you will follow them around. More caution is needed when filming children, as their parents may be more wary of their privacy.