The simulation above is a Java applet that simulates the scanning of a raster display. Move the CRT wireframe (by clicking and dragging) in order to get a better feel.
There were a few attempts at building systems with downloadable or programmable character generators. And a few systems added an extra byte to specify the foreground and background colors of the character cell. Lots of tank/maze arcade games in the 70's worked this way. But by the late 70's and early 80's the price of memory started a free-fall and the graphics terminal was born. In a later lecture we'll go into a lot more detail about the notion of a framebuffer an how it is fundamental to modern computer graphics.