Our proximity to the sun would matter more if it heated us via convection. Like a hot oven, convection relies on a medium like air to carry heat to the target. But space is a vacuum, and without a gas or liquid to carry convection heat, the sun has to use radiant heat. Electromagnetic waves carry energy that heats molecules of air and earth on their arrival, rather than being hot when they arrive and transferring that heat. It’s the same kind of heating that makes a bonfire so hot on your face but not on your back—the fire isn’t warming the air, it’s sending out energy waves that causes your skin to heat up. The sun is over 90 million miles away at any given time, so adding or subtracting 3 million clicks doesn't make a difference in radiant heat that we can notice.