Found this after too much Googling so listing here for future reference.

Stop X Starting at Boot on an Upstart controlled system

update-rc.d -f remove gdm doesn't work:
That is because Ubuntu doesn't use System V init files for startup any more, it uses a program called Upstart. You have to edit the file /etc/init/gdm.conf and comment the 'start' line to obtain something like:

Code:
#start on (filesystem
#          and started dbus
#          and (graphics-device-added fb0 PRIMARY_DEVICE_FOR_DISPLAY=1
#               or drm-device-added card0 PRIMARY_DEVICE_FOR_DISPLAY=1
#               or stopped udevtrigger))
stop on runlevel [016]

http://www.uluga.ubuntuforums.org/showpost.php?p=9691917&postcount=2