The "pixellizing" effect you're seeing is called aliasing. It is an artifact of using a digital medium—you cannot get a truly smooth angled or curved line when the image is built of discrete pixels that are either "on" or "off." The solution is to use some intermediate shades in places along the boundaries to create the illusion of smoothness. This is called anti-aliasing. For reference, I posted some example images a while ago here: http://www.cartographersguild.com/sh...ll=1#post14957
Most of the time, you want to start with anti-aliased lines because there's not really any easy way to reliably and accurately anti-alias them later. To that end, avoid using the Pencil tool, and be careful not to threshold things too harshly. Also, work at a resolution high enough that you are not relying on pixel or subpixel accuracy for your detail.
Now, if you really need to fake antialiasing, there are a number of approaches you can use. Personally, I like to use some resizing tricks. In the Preferences > General menu, set Image Interpolation to Bicubic Smoother. Enlarge your image to 200% in both directions. Give it a 1 pixel Gaussian blur. Go back to the Preferences and set Image Interpolation to Bicubic Smoother. Reduce your image to 50% in each direction (returning it to the original size). The lines should be somewhat smoother. Try it with and without the blur step, try different combinations of the interpolation algorithms (except nearest neighbor—that one will actually enhance the aliasing instead of reducing it), and see if going to 400% or 800% and back down does anything more for you. Different sizes of blur will give you different results, too.
This is not truly antialising, though, as it softens the lines rather than blending them properly. You could also try some tricks with layer effects. A 1-pixel stroke or even a very small outer glow might do the trick for some shapes and styles.
Here's a sample of what I did while I was experimenting. In the upper left is my original, aliased shape. In the top center, I simply increased the size by 200% then reduced it by 50% with Bicubic interpolation both directions. Top right, I did the same thing with a 1-pixel Gaussian blur. Bottom left, I used the resizing process I described above. Bottom center has an outer glow set to Blend Mode: Multiply, 65% opacity, Color: Black, Technique: Softer, 0% spread, 1 px size, and a linear contour.
Untitled-1.png