It's structurally limited, in practice, just as free speech is. It's part of becoming a member of a group that you lose perfect freedom, in any respect. Part of what defines every group is how you lose those freedoms - but most people would rather think of it as a sacrifice or civil submission rather than a loss.