Update:  Also this option from No Machine: How to use NoMachine 4 on a headless Linux server

On a Digital Ocean CentOS 6.x VPS

I did the following to get an X environment running

yum groupinstall Xfce
yum groupinstall "X Window System"

set the id:3:initdefault: line  in    /etc/inittab

to id:5:initdefault:

and run to start X

init 5

I installed noMachine

yum localinstall ./nomachine_4.5.0_1_x86_64.rpm

When I connected from the noMachine client the resolution was limited to 1024×768

So first turn off X

init 3

Then as root run

X -configure

Which will write a file to /root/xorg.conf.new

Then copy to the default location

cp /root/xorg.conf.new /etc/X11/xorg.conf

Then add a new horizontal refresh and sync settings to the monitory area

Section "Monitor"
        Identifier   "Monitor0"
        VendorName   "Monitor Vendor"
        ModelName    "Monitor Model"
        # add these two lines
        HorizSync 31.5 - 70.0
        VertRefresh 50-160
EndSection

After making the above change and then running init 5 I could run xrandr and it would list a resolution higher than 1024×768 in this case it was 1280×1024

xrandr
xrandr: Failed to get size of gamma for output default
Screen 0: minimum 640 x 480, current 1280 x 1024, maximum 1280 x 1024
default connected 1280x1024+0+0 0mm x 0mm
   1280x1024      61.0* 
   1024x768       85.0  
   800x600        86.0  
   640x480        86.0

The problem (I think) was a combination of lack of ram and no settings for the VertRefresh/HorizSync settings. I saw these errors in Xorg.0.log

[1379954.435] (II) CIRRUS(0): Not using default mode "640x350" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "320x175" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x400" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "320x200" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "720x400" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "360x200" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "320x240" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x480" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "320x240" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x480" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "320x240" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x480" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "320x240" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "400x300" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "400x300" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "400x300" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (vrefresh out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "400x300" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (hsync out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "400x300" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1024x768i" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "512x384i" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "512x384" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1024x768" (hsync out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "512x384" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1024x768" (hsync out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "512x384" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1024x768" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "512x384" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1152x864" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "576x432" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1280x960" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x480" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1280x960" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x480" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1280x1024" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x512" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1280x1024" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x512" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1280x1024" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "640x512" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1600x1200" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1600x1200" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1600x1200" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1600x1200" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1600x1200" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "800x600" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1792x1344" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "896x672" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1792x1344" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "896x672" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1856x1392" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "928x696" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1856x1392" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "928x696" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1920x1440" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "960x720" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1920x1440" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "960x720" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "832x624" (hsync out of range)
[1379954.435] (II) CIRRUS(0): Not using default mode "416x312" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1400x1050" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "700x525" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1400x1050" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "700x525" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "1920x1440" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "960x720" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "2048x1536" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "1024x768" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "2048x1536" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "1024x768" (bad mode clock/interlace/doublescan)
[1379954.435] (II) CIRRUS(0): Not using default mode "2048x1536" (insufficient memory for mode)
[1379954.435] (II) CIRRUS(0): Not using default mode "1024x768" (bad mode clock/interlace/doublescan)
[1379954.435] (**) CIRRUS(0): *Default mode "1024x768": 65.0 MHz, 48.4 kHz, 60.0 Hz
[1379954.435] (**) CIRRUS(0): *Default mode "800x600": 40.0 MHz, 37.9 kHz, 60.3 Hz
[1379954.435] (**) CIRRUS(0): *Default mode "800x600": 36.0 MHz, 35.2 kHz, 56.2 Hz
[1379954.435] (**) CIRRUS(0): *Default mode "640x480": 25.2 MHz, 31.5 kHz, 59.9 Hz