PS: Question on huge size images

[Redrobes]

Are you running a 64 bit version of Gimp ? Cos at 20 layers and maybe some hidden undo buffers perhaps its getting close to the 3-4Gb limit of 32bit apps. If it were me, id run up the sysinternals process viewer and see how much memory and how many handles etc its burning with that map open. You can look at the pagefile delta and see if its spending all its time page file thrashing too. Well, a topic for another thread but something is not quite right there. I guess being on topic tho it is worth ensuring that your system is tuned up to cope with big maps if your heading into a big map job. Just seems odd to me that PS and Gimp don't handle large images very well. I thought PS had a large image mode or is that just for the file save options. With all the resources of Adobe, cant they figure it out ? What do the pro design outfits do when customer asks for a huge billboard with photos on it up close like in an expo or something ?

[Djekspek]

Guess I'm not the only one struggling with big files. All thanks so much for the input so far. (and you're right RR, i still need to really testdrive your ViewingDale software, shame on me ) ... I'm afraid that for this I wont have time to learn another tool (like viewingdale or a vectorish program). I did some tuning and it now works with 3 to 4 layers at 300dpi, but after that, things start going down hill fast. As far as I can tell it appears to be a CS3 (+32 bit?) thing in that it can only use about 3GB of my 6GB ram. My CPU is almost asleep but my memory is overpacked (at least, that's what these little graphs of my 'resource manager' tell me ... The saved PS-file itself was not even 3GB so i would think it could run completely in memory... hmm... very confusing all this to me ... So i guess it's either upgrading to 64bit CS4/5 and/or buying a fast external drive ... to shoot an elephant, one needs a canon, it seems... cheers

[Antheon]

I have CS3 myself, so I looked for you. Here is what I got:

Allocating memory above 2 GB with 64-bit processors

Photoshop CS3 is a 32-bit application. When it runs on a 32-bit operating system, such as Windows XP Professional and some versions of Windows Vista, it can access the first 2 GB of RAM on the computer.The operating system uses some of this RAM, so the Photoshop Memory Usage preference displays only a maximum of 1.6 or 1.7 GB of total available RAM. If you are running Windows XP Professional with Service Pack 2, you can set the 3 GB switch in the boot.ini file, which allows Photoshop to use up to 3 GB of RAM.

Important: The 3 GB switch is a Microsoft switch and may not work with all computers. Contact Microsoft for instructions before you set the 3 GB switch, and for troubleshooting the switch. You can search on the Microsoft support page for 3gb for information on this switch.

When you run Photoshop CS3 on a computer with a 64-bit processor (such as a, Intel Xeon processor with EM64T, AMD Athlon 64, or Opteron processor) running a 64-bit version of the operating system (Windows XP Professional x64 Edition or Windows Vista 64-bit) and with 4 GB or more of RAM, Photoshop will use 3 GB for it's image data. You can see the actual amount of RAM Photoshop can use in the Let Photoshop Use number when you set the Let Photoshop Use slider in the Performance preference to 100%. The RAM above the 100% used by Photoshop, which is from approximately 3 GB to 3.7 GB, can be used directly by Photoshop plug-ins (some plug-ins need large chunks of contiguous RAM), filters, or actions. If you have more than 4 GB (to 6 GB), then the RAM above 4 GB is used by the operating system as a cache for the Photoshop scratch disk data. Data that previously was written directly to the hard disk by Photoshop is now cached in this high RAM before being written to the hard disk by the operating system. If you are working with files large enough to take advantage of these extra 2 GB of RAM, the RAM cache can speed performance of Photoshop. Additionally, in Windows Vista 64-bit, processing very large images is much faster if your computer has large amounts of RAM (6-8 GB).

The default RAM allocation setting is 55%. This setting should be optimal for most users. To get the ideal RAM allocation setting for your system, change the RAM allocation in 5% increments and watch the performance of Photoshop in the Performance Monitor. You must quit and restart Photoshop after each change to see the change take effect.

The available RAM shown in the Performance preference automatically deducts an amount that is reserved for the operating system from the total RAM in your computer. You shouldn't set the percentage of RAM to be used by Photoshop to 100% because other applications which run at the same time as Photoshop (for example, Adobe Bridge) need a share of the available RAM. Some applications use more RAM than you might expect. For example, web browsers can use 20-30 MB of RAM, and music players can use 20-50 MB RAM. Watch the Performance Monitor to view the RAM allocations on your computer.

Watch your efficiency indicator while you work in Photoshop to determine the amount of RAM you'll need to keep your images in RAM. The efficiency indicator is available from the pop-up menu (choose Show > Efficiency) on the status bar of your image and from the Palette Options on the Info Palette pop-up menu. When the efficiency indicator goes below 95-100%, you are using the scratch disk. If the efficiency is around 60%, you'll see a large performance increase by changing your RAM allocation or adding RAM.

EDIT:
Hello mind, are you there somewhere? *sigh*
Maybe I should drop the link for you, guys ...
Click me!


Antheon

[Djekspek]

Antheon, that's great info! I had seen it but didnt understand ... I tried understanding it again (and I still dont ) and I somehow 'tweaked' the 'RAM thingie' and now have it running with 10 layers and it's pretty smooth have some rep! cheers!

[Antheon]

I'm glad to hear it was of help. If you haven't looked at the link, I suggest you to do it because the page is about CS3 handling large files with further tuning tips. : )


Antheon

[Natai]

I've been working on a large map for a pen&paper RPG I've been designing. I'll have to check the specs when I get home, but in my experience the amount of RAM in relation to the saved size of the file has made a huge difference, as has 64-bit. A previous PSD version of the map on a computer with 2-4GB of RAM and PS CS3 would take at least 30 min to open and a good hour to save. Everything was slow and jittery like so many of you have described.

I started from scratch for my current iteration, and this file also marks the first time I discovered the 4GB file-size limit for PSD files. As I said, I'll need to check the exact dimensions, but I believe the image is somewhere around 37800x21600 (42"x24" @ 900px/in) with at least 30-40 layers. My new computer (Core i7 920, 6GB DDR3 RAM, 2x 1TB SATA HDD, PS CS4, Win7 x64) handled it with no problem. Everything is smooth, opening took around 2min and saving around 8-10min. The PSB file was over 7GB, so it would not all fit in memory. I recently upgraded to 12GB of RAM and that has shown a marked improvement. The ever-growing 7.5+ GB file opens in 1-2min, saves in about 5-6min, and everything runs smoothly. PS is generally using 7.5-8.5GB of memory in the resource monitor. I typically have several instances of IE, Word, Excel, and a video playing on a second monitor while working on this.

My suggestion for anyone working with large PS files would be to make sure that the amount of RAM available to PS is at least equal to the size of the saved file. This may mean increasing the amount of RAM as well as upgrading to a 64-bit version of your OS and PS. Of course, you also want to make sure you've got a fair amount of free space for your scratch disk.

[tilt]

otherwise djespek, sketch the whole thing up and divide it into 4 or more documents for the fine details, then flatten the layers before putting the results together in the end. And if you think of changing stuff, my bet would be on changing to windows 7 instead of Vista - cause that takes a lot less resources (using Vista myself but thinking of upgrading when I get rich *lol*)

[Redrobes]

Hi Natai & welcome to the guild. Good info there.

Heres the bottom line. Computers run with (nowadays) 32bit or 64bit. To be fully 64bit you need a) a 64 bit OS - Windows XPx64 Vista64 or 7(64bit). All the Linuxes have been 64bit for years. Not sure about Macs. You also need b) a 64bit CPU which is required for a 64bit OS and c) The app your running has to be 64bit. If you have all three then your 64 bit.

With 32 bit all your memory pointers are 32bit. This means that 1^32 = 4Gb. I.e. you cannot point to memory > 4Gb because you have run out of indexes. Now these pointers are virtual RAM not physical RAM. I wont go into that much but the effect is that EACH app can access up to 4Gb so you can blow 4Gb out of a 6Gb system and run out and yet still play some video and browse the web with another app. If you run out of memory in an app it pages it off to the harddrive. You can in theory install a RAM drive and point your pagefile at the ramdrive and then the app will page it out to a different apps 4Gb of space so one app is kinda using more than 4Gb. But lets face it this is all a whole lot of hassle and you should not install more than 4Gb on a 32bit system.

With 64bit the pointers are 64bit and therefore absolutely huge - like count of atoms in universe type numbers so its never going to run out. Each app can now access as much RAM as you can get into the PC. There is no limit any more.

Your HDD is very very slow and, being generous, typically has a sustained write speed of about say 50Mb per sec. So 1Gb takes about 20secs and so about 3Gb per min at full sustained maxed out rate. Therefore if you have 6 or 8 Gb of RAM there is not much point in having a pagefile cos your system hangs for multiple tens of seconds when it must page out. So with small amount of RAM systems having a pagefile is ok, multi gig RAM systems and theres no point. Compare with RAM write speed at a few gig per second. Theres always been about a 100:1 speed ratio between ram and HDD.

The PSD file is very likely to be compressed like a PNG. I dont know this for sure but as a rule its quicker to compress with the CPU and save the smaller data to HDD than not to compress and save that time but wait longer for the HDD to save it. Therefore I would guess that the amount of RAM used by an image set with all its layers would be bigger than the PSD file. Perhaps as a rule having RAM = PSD file size is not such a bad thing. My suggestion tho is to run the process viewer and see how much RAM its using.

So heres my top tips. If in 32 bit land get 4Gb or so and no more. If you want to go bigger then preferably upgrade to 64 bit bit if not then either get more ram and install a RAM drive and put all swap data onto the RAM drive. Or, get some super fast 10,000 rpm drives and RAID them into a striped array and try to get the read/write speed up and use that as the pagefile drive. Or if you have the dosh, get solid state flash drives and use them for that job. Upgrading is easier tho

Or change to an app which doesn't use memory like PS does...

[tilt]

thats a good run down redrobes - very nice... defintly makes me think about the specs for my next machine ... in about 2020 or so...
concerning photoshop - the rule of thumb I learned some years ago is that it uses about 3 times the ram as the size of the image, so if you have a 1 gig image, it uses 3 gigs to handle it.

[Natai]

Thanks for the welcome, Redrobes.
Okay, my file specs were definitely off a bit. The image is 20122x11634 @ 450 px/in. Apparently the file size was way off too, I must have been thinking about the old version of the file. The PSB file is about 3.7GB saved, and PS consumes around 7.75GB in memory while I'm working with it. So tilt's point is a good one - go with 2-3 times the amount of RAM relative to the saved file size.

Definitely a good explanation of the 64-bit and RAM limitations. A couple of other points I seem to remember from when I was researching prior to transitioning from 32-bit to 63-bit Vista: because of how the pointers work the most RAM any given application can use on a 4GB 32-bit system will probably be in the neighborhood of 3.2 to 3.7GB. Most 32-bit OS will only actually detect 3.7 to 3.8GB of available RAM, depending on your setup. Sometimes the type and configuration of your video card will also affect this, as some setups allow the video card to use a portion of system RAM as a sort of virtual VRAM. So, bottom line if you're working with files larger than about 1.5GB, you will probably benefit from upgrading to 64-bit. Just be sure to keep in mind what Redrobes posted - you're entire system has to be 64-bit, the CPU, OS and the application.

For the most part (with the exception of the VRAM and memory pointer issue) the video card has very little impact. CS4 enables GPU support to increase draw speeds on the monitor, but I think that's about it.
However, that may be changing now. The new DX11 cards enable the computer to offload computing tasks to the graphics card. This isn't something CS4 is really capable of taking advantage of, but I've heard CS5 might be able to utilize this feature. I'm not sure how much of a performance boost you would see, though it might not be that great if only the GPU itself is used as PS doesn't really use that much CPU power. It would probably be of greater benefit if this setup allowed PS to utilize the VRAM on the card. Anybody tried anything like to see if it makes a difference? And then there's the issue of cost: upgrading to CS5 aside, the new nvidia DX11 card start at over $400. Upgrading to 64-bit with more RAM is definitely a better option.