Yeah this is where it's tricky, can time exist without it 'passing' ? Can we really dissociate 'passage of time' and time like that ? Intuitively no, and this is why I find it better to compare passage of time with moving through space ...
In that case we are not in disagreement.
If we say that passage of time is like passage through space, then to the extent that this is true and meaningful (and I have my doubts about that) we have to say that time is the analog of space.
But past time being infinite, means that it is no longer just 'potential' but reality. It means I have actually lived on forever.
No, it doesn't. There is no way in which this relates to you or me
personally. The proposition that past time is infinite does not conflict with the statement that I was born in 1974 any more than the proposition that 1973 existed conflicts with that same statement. Why would it?
The proposition that the past is infinite does not contradict the proposition that certain specific things had beginnings.