I'm amazed this thread hasn't turned into a bloodbath of opinions, half-baked ideas, and internet factoids
The basic starting point is that your house itself will lose more heat the warmer it is, so it makes sense for it not to be warmer than necessary.
Warmer than necessary is difficult to define. Obviously when you're home it is fairly easy to say how warm it needs to be, but how cool should it be allowed to get at other times? Allowing the masonry to become too cool can lead to condensation although just how cool varies a lot. A small house full of kids and wet washing might have warm walls running with water while a larger one less occupied can have walls cold to touch without problems. The type and amount of insulation will also make a big difference. If you wonder how we used to cope when you and I were kids, partly we just used to accept damper houses and partly they had a lot more "ventilation", both deliberate and unintentional. Partly, many of the rooms with cold walls just never got heated!
Most of the energy stored in a house is in the solid objects, walls, floors, and fittings, while the air only contains a small fraction of the heat. This means that a house that has thoroughly cooled down can be difficult to heat up again to comfortable levels. The thermostat may say the air is at 20C but you still feel cold because the structure of the house and the furniture is considerably cooler. You might have noticed the effect when you came back from holiday, and not just because you got used to it being 30C in Ibiza
So letting the walls cool down will reduce the heat lost through them, but if it causes you to thrash the central heating afterwards you might not gain much from it.
There are other reasons not to let your house get too cold in between needing it warm, related to the way modern heating systems work. Condensing boilers and radiators are most efficiently operated for relatively long periods at relatively low settings, which isn't practical if your house is at 10C when you come home from work. Underfloor heating simply can't raise the temperature of a house quickly even if you were prepared to turn the boiler up. Remember though, that you are talking about an efficiency difference of just a few percent so don't go mad and assume your radiators have to be ticking over 24/7. Also, once the desirable power output drops low enough that the boiler has to cycle you'll lose that percent or two of efficiency, and some would say a lot more besides
This used to be simple, you manually or on a timer turned the heating on when you were in and off at other times. Now the trend in controls is for heating to be "on" 24 hours a day, although with a setback temperature during the night and usually during the middle of the day. Typically the setback temperature will be low enough that the heating won't come on at all at these times because a well-insulated house won't cool down fast enough to reach the setback temperature, so not so different to a simple timer. This has almost reached "mantra" levels with supposedly learned people claiming it is most efficient to keep your house at a constant temperature 24/7. Don't get carried away with this idea. It can work well if you're home for a large fraction of the time and need the whole house to be heated. Otherwise let it cool down when you're not there, especially since you can time the heating to come on gradually an hour or two before you need it. And look into whether you can heat different parts of the house at different times. For example, how much of the house do you need heated to get dressed and grab a bagel in the morning, and how often do you use that box room?