I would want to have the full picture on the planning history of both plots before I got into a fight with the planners.
Agreed - worth checking. If it is how the OP portrays, it's not reasonable to enforce and now due to the expiration of 10 years it is not lawful to enforce.
If a condition is not met, then the permission could be said to have not been fully enacted, and so the breach would not have a time limit.
It was either implemented, in which case the time (10 years) has expired. If it was not enacted, then it would not technically benefit from planning permission. In which case the development would have need to have been built for a period of 4 or more years (which is the case in the matter) and would be lawful through that route.
It is worth a double check on the conditions in case they had some specific wording (or associated timeframe). Hopefully it wont get that far, but if it should the condition shouldn't be enforceable on either counts.