The nature of potential is that the zero point is arbitrary; it can be set like the origin of a coordinate system. That is not to say that it is insignificant; once the zero of potential is set, then every value of potential is measured with respect to that zero. Another way of saying it is that it is the change in potential which has physical significance. The zero of electric potential (voltage) is set for convenience, but there is usually some physical or geometric logic to the choice of the zero point. For a single point charge or localized collection of charges, it is logical to set the zero point at infinity. But for an infinite line charge, that is not a logical choice, since the local values of potential would go to infinity. For practical electrical circuits, the earth or ground potential is usually taken to be zero and everything is referenced to the earth.