In many cultures, including ours, pointing at other people is considered rude because it’s associated with blame allocation (‘to point the finger at…’). Also, by pointing at someone, you automatically, and without their consent, make them an object of scrutiny.
