What do I want? A minimal install! How do I get it? Through apt-get!

apt-get install xserver-xorg-core gdm3 gnome-core

And you have yourself a nice minimal desktop, great for use on servers.