A rural Herefordshire bungalow has not been lived in by a farm worker for two decades, in breach of a planning condition that its owners say is now unenforceable.