Yes. E=mc2. That applies to the batteries too.
Where does that mass go? It doesn't teleport from the battery to the photon. It travels with the electrons which the battery put into a higher energy (and thus higher mass) state. These electrons then lose mass at the LED diode when emitting the photon. So the creation of the photon still involves an electron losing mass at the LED.
Close but not completely correct.
Let us assume the following circuit:
The plus/positive battery pole/clamp (1) is connected to switch input pole/clamp (2) with a wire.
The switch output pole/clamp (3) is connected to diode + anode (4) with a wire.
The minus/negative battery pole/clamp (5) is connected to - cathode (6) with a wire.
When the switch is off the voltage delta is between switch input pole (2) and - cathode (6).
When the switch is on the voltage delta is between + anode (4) and - cathode (6).
The energy from battery travels along the wire through EM field.
The voltage delta will excite an electron from the - cathode (6) tip that is available for the flow.
The first electron that would emit light is already at the - cathode (6) tip when the switch is turned on.
The first electron would not come from the battery.
Assuming only one photon emitted the light between on/off change then energy from the battery gave energy through the EM field to the electron at the - cathode (6) tip to become free, to cross the cavity.
The electron does 'Bremsstrahlung' at the end of cavity, the electron changes its kinetic energy for the photon.
In conclusion the energy came to the LED from the battery and the mass is missing from the battery at the center of the flashlight.
The mass is not missing at the tip.
To think about it differently.
There is our setup from the above.
We turn on the switch and we let it go for a day.
We turn off the switch.
Where is the 'mass hole'?
... in the battery, center of the flashlight, there is no 'mass hole' at the LED.
Why this would not apply for one electron and photon?