Hello jonc and gyra,
Cheap to buy, maintain and run at well under a grand for a good 'un?
Appreciating classic?
Learner legal?
An early '70's Honda CB125S will tick all your boxes.
Look for the best, cleanest (no rust or rot, look under the dust), most original one you can find, take someone who really knows what to look for when buying a bike, and when you find it make it good and keep it.
Spares are not too difficult to get, and quite cheap.
When you are ready for a bigger bike, keep the wee Honda. You probably won't want to sell it anyway.
Another little bike which may suit is the CG 125 Honda, which although a sibling of the CB model and a "classic" in its own right, is fundamentally different and does not carry the same status in the eyes of many of the classic bike fraternity. It would, however, be readily passed on to aid the purchase of a more desirable mount after successfully clearing DOT hoops.
Alternatively, borrow, make roadworthy and road legal, something to practice on; and do the test on a school bike saving much of your budget for something larger.
There is a possibility that 250cc machinery over a certain age may be used by learners, but if so you should bear in mind that larger machines with more than one cylinder will be correspondingly more expensive on all counts.
I wish you and gyra all the best of luck.
Keep between the hedges
A.