In Einstein's Relativity, spacetime is a thing. You can move in space but move in time too. So a 'straight line' can put you in the same place but at a different time. That said, I suspect Dr. Siegel could tell us a lot more about this interesting observation you are making.

You have mass so time is inescapable to you. If you didn't you would know time or space or spacetime. If you were a photon even "straight line" is giving reality too much credit. Photons are simply emitted and absorbed by different masses that cannot occupy the same spacetime coordinates in no time at all! -- by the equations.

https://medium.com/liecatcher/photons-do-not-know-space-or-time-or-even-gravity-dfba0b45dcdf