Here's a fairly simple fantasy forest styling effect in QGIS:
effect-based-forest.png

The neat thing is that the underlying polygon can be edited and no restyling needs to be done.

Here's an alternate version of the same concept:
AlternateStyleTutbad.png

The tree trunks are an offset marker outline. Then the interior foliage effect consists of a bunch of randomly placed little circles with outlines, underneath a smaller number of larger circles without outlines.
There are a lot of circles, so it takes a while for the layer to render any time the canvas is updated, but QGIS seems to handle it pretty gracefully.