Hah, cool, I'm working on one right now too. Got most of the way through it, it's an awesome learning experience about what you can achieve with FPGAs and what the limitations are. Especially implementing the 'complex' instructions like drawing.
The way it's usually done now is, people simply go with implementing some kind of a PSM ("programmable state machine"), which is essentially a tiny microprocessor, and then let the program do the, say, drawing (by either setting bits in video memory, sending commands to an SPI display, or directly generating VGA signals).
It gets the job done and can be efficient from a development perspective. If high speed (ops/sec) operation isn't required, this sort of approach can be pretty useful.
I needed to do a "headless" build of an FPGA where the software wasn't ready yet, but the RF guys needed the FPGA operate as if software was there and running the show. I did a sort of PSM as you call it that was like a really basic processor. Peek, Poke, Test, Jump, Inc, Dec. It was very helpful for debug, and the RF guys were able to make progress weeks before software was ready to go. Plus the software builds were very focussed on flight, and not doing special configurations to help others figure out corner case issues.
For a hobby project, depending on the goals of why one is doing the project, I think it's a fine approach. Gets results fast which helps with motivation and momentum to learn more difficult things. It's also a natural bridge for people coming from a software background.
