Building Inspectors, whether private or LABC, are only allowed to enforce the Building Regulations insofar as they affect health and safety (except for a few minor exceptions).
If a beam is not the "right" size, or is missing, it does not mean that the structure is substandard and might fail. Legally, it simply means that your builder may have done it a different way and not necessarily in accordance with the plans.
The problem in complaining about inspectors is that most Building Regulations are now cast as 'performance standards', rather than by prescriptive rules. It is therefore often very difficult to prove that something does not comply with the regs.
You have been unlucky with your builder, and probably only slightly unlucky with your inspector; there are good and poor inspectors on both sides of the divide.
Put it down to experience; personally, I wouldn't waste time trying legal redress.