There have been a couple of reports of this happening, and the solution each time was a problem with the spec and the version of the designer. If you edit and save a specification with a newer version of the check designer, one or more features that an earlier version may not support could have been saved with it. When you then try to print using that spec, using an older version of the designer, it would see a feature it can't support and give you an error message.
How did you create the specification? Was it edited from one of the sample specs, or did your dealer create it for you in their office, or did you create it from scratch, etc?
You may also want to uninstall and reinstall the program, just in case something happened to your version of the designer program.
Your steps for defining the custom size of the check form is correct. That is not the cause of the errors.