Hi everyone. For years, I've been using ssh -X just fine to connect to our various remote machines and interact with X applications on the remote machines via the display in front of me, but recently I installed a fresh CentOS 6.3 x64 on real hardware and it isn't working. The DISPLAY environment variable is never populated on the connected session and even manually setting it fails. "hope" is my local machine, "dy" is an established CentOS 5.x machine that works fine and "dz" is the problematic one. dz was installed with "X Windows System" and indeed on the console, X works just fine and I have a Gnome desktop too. [larocque@hope ~]$ echo $DISPLAY :0 [larocque@hope ~]$ [larocque@hope ~]$ ssh -Xvv root [ at ] dz [ dot ] cms [ dot ] math [ dot ] ca OpenSSH_5.6p1, OpenSSL 1.0.0j-fips 10 May 2012 debug1: Reading configuration data /home/larocque/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to dz.cms.math.ca [10.5.7.201] port 22. debug1: Connection established. debug1: identity file /home/larocque/.ssh/id_rsa type -1 debug1: identity file /home/larocque/.ssh/id_rsa-cert type -1 debug1: identity file /home/larocque/.ssh/id_dsa type -1 debug1: identity file /home/larocque/.ssh/id_dsa-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.6 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa-cert-v01 [ at ] openssh [ dot ] com,ssh-dss-cert-v01 [ at ] openssh [ dot ] com,ssh-rsa-cert-v00 [ at ] openssh [ dot ] com,ssh-dss-cert-v00 [ at ] openssh [ dot ] com,ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc [ at ] lysator [ dot ] liu [ dot ] se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc [ at ] lysator [ dot ] liu [ dot ] se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 [ at ] openssh [ dot ] com,hmac-ripemd160,hmac-ripemd160 [ at ] openssh [ dot ] com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 [ at ] openssh [ dot ] com,hmac-ripemd160,hmac-ripemd160 [ at ] openssh [ dot ] com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib [ at ] openssh [ dot ] com,zlib debug2: kex_parse_kexinit: none,zlib [ at ] openssh [ dot ] com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc [ at ] lysator [ dot ] liu [ dot ] se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc [ at ] lysator [ dot ] liu [ dot ] se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 [ at ] openssh [ dot ] com,hmac-ripemd160,hmac-ripemd160 [ at ] openssh [ dot ] com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 [ at ] openssh [ dot ] com,hmac-ripemd160,hmac-ripemd160 [ at ] openssh [ dot ] com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib [ at ] openssh [ dot ] com debug2: kex_parse_kexinit: none,zlib [ at ] openssh [ dot ] com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found hmac-md5 debug1: kex: server->client aes128-ctr hmac-md5 none debug2: mac_setup: found hmac-md5 debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug2: dh_gen_key: priv key bits set: 129/256 debug2: bits set: 513/1024 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host 'dz.cms.math.ca' is known and matches the RSA host key. debug1: Found key in /home/larocque/.ssh/known_hosts:153 debug2: bits set: 533/1024 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: general ssh key for larocque at CMS (larocque [ at ] cms [ dot ] math [ dot ] ca) (0x7f599d07efb0) debug2: key: /home/larocque/.ssh/id_rsa ((nil)) debug2: key: /home/larocque/.ssh/id_dsa ((nil)) debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug2: we did not send a packet, disable method debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information Credentials cache file '/tmp/krb5cc_500' not found debug1: Unspecified GSS failure. Minor code may provide more information Credentials cache file '/tmp/krb5cc_500' not found debug1: Unspecified GSS failure. Minor code may provide more information debug1: Unspecified GSS failure. Minor code may provide more information debug2: we did not send a packet, disable method debug1: Next authentication method: publickey debug1: Offering RSA public key: general ssh key for larocque at CMS (larocque [ at ] cms [ dot ] math [ dot ] ca) debug2: we sent a publickey packet, wait for reply debug1: Server accepts key: pkalg ssh-rsa blen 646 debug2: input_userauth_pk_ok: SHA1 fp 9c:0c:da:c8:f0:4f:64:cd:59:27:d3:3f:a4:72:c2:fd:cc:63:9b:83 debug1: Authentication succeeded (publickey). Authenticated to dz.cms.math.ca ([10.5.7.201]:22). debug1: channel 0: new [client-session] debug2: channel 0: send open debug1: Requesting no-more-sessions [ at ] openssh [ dot ] com debug1: Entering interactive session. debug2: callback start debug2: x11_get_proto: /usr/bin/xauth list :0 2>/dev/null debug1: Requesting X11 forwarding with authentication spoofing. debug2: channel 0: request x11-req confirm 0 debug2: client_session2_setup: id 0 debug2: channel 0: request pty-req confirm 1 debug1: Sending environment. debug1: Sending env XMODIFIERS = @im=none debug2: channel 0: request env confirm 0 debug1: Sending env LANG = en_CA.utf8 debug2: channel 0: request env confirm 0 debug2: channel 0: request shell confirm 1 debug2: fd 3 setting TCP_NODELAY debug2: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel_input_status_confirm: type 99 id 0 debug2: PTY allocation request accepted on channel 0 debug2: channel 0: rcvd adjust 2097152 debug2: channel_input_status_confirm: type 99 id 0 debug2: shell request accepted on channel 0 Last login: Tue Oct 2 11:40:38 2012 from hope.ott.cms [root@dz ~]# [root@dz ~]# [root@dz ~]# [root@dz ~]# [root@dz ~]# echo $DISPLAY [root@dz ~]# xclock Error: Can't open display: [root@dz ~]# [root@dz ~]# [root@dz ~]# [root@dz ~]# [root@dz ~]# grep X /etc/ssh/*_config /etc/ssh/ssh_config:# ForwardX11 no /etc/ssh/ssh_config:# If this option is set to yes then remote X11 clients will have full access /etc/ssh/ssh_config:# to the original X11 display. As virtually no X11 client supports the untrusted /etc/ssh/ssh_config: ForwardX11Trusted yes /etc/ssh/ssh_config: SendEnv XMODIFIERS /etc/ssh/sshd_config:AcceptEnv XMODIFIERS /etc/ssh/sshd_config:#X11Forwarding no /etc/ssh/sshd_config:X11Forwarding yes /etc/ssh/sshd_config:#X11DisplayOffset 10 /etc/ssh/sshd_config:#X11UseLocalhost yes /etc/ssh/sshd_config:# X11Forwarding no [root@dz ~]# [root@dz ~]# [root@dz ~]# [root@dz ~]# rpm -qa|grep -i x11|sort ConsoleKit-x11-0.4.1-3.el6.x86_64 dbus-x11-1.2.24-7.el6_3.x86_64 libX11-1.3-2.el6.x86_64 libX11-common-1.3-2.el6.noarch pulseaudio-module-x11-0.9.21-14.el6_3.x86_64 qt-x11-4.6.2-24.el6.x86_64 xorg-x11-apps-7.4-10.el6.x86_64 xorg-x11-drivers-7.3-13.3.el6.x86_64 xorg-x11-drv-acecad-1.5.0-3.el6.x86_64 xorg-x11-drv-aiptek-1.4.1-2.el6.x86_64 xorg-x11-drv-apm-1.2.3-2.el6.x86_64 xorg-x11-drv-ast-0.91.10-1.el6.x86_64 xorg-x11-drv-ati-6.14.2-9.el6.x86_64 xorg-x11-drv-ati-firmware-6.14.2-9.el6.noarch xorg-x11-drv-cirrus-1.3.2-2.el6.x86_64 xorg-x11-drv-dummy-0.3.4-1.el6.x86_64 xorg-x11-drv-elographics-1.3.0-2.el6.x86_64 xorg-x11-drv-evdev-2.6.0-2.el6.x86_64 xorg-x11-drv-fbdev-0.4.2-2.el6.x86_64 xorg-x11-drv-fpit-1.4.0-2.el6.x86_64 xorg-x11-drv-glint-1.2.5-1.el6.x86_64 xorg-x11-drv-hyperpen-1.4.1-2.el6.x86_64 xorg-x11-drv-i128-1.3.4-1.el6.x86_64 xorg-x11-drv-i740-1.3.2-2.el6.x86_64 xorg-x11-drv-intel-2.16.0-4.el6.x86_64 xorg-x11-drv-keyboard-1.6.0-1.el6.x86_64 xorg-x11-drv-mach64-6.9.0-1.el6.x86_64 xorg-x11-drv-mga-1.4.13-7.el6.x86_64 xorg-x11-drv-mouse-1.7.0-4.el6.x86_64 xorg-x11-drv-mutouch-1.3.0-2.el6.x86_64 xorg-x11-drv-nouveau-0.0.16-13.20110719gitde9d1ba.el6.x86_64 xorg-x11-drv-nv-2.1.18-2.el6.x86_64 xorg-x11-drv-openchrome-0.2.904-4.el6.x86_64 xorg-x11-drv-penmount-1.5.0-2.el6.x86_64 xorg-x11-drv-qxl-0.0.14-13.el6_2.x86_64 xorg-x11-drv-r128-6.8.1-3.el6.x86_64 xorg-x11-drv-rendition-4.2.4-1.el6.x86_64 xorg-x11-drv-s3virge-1.10.4-2.el6.x86_64 xorg-x11-drv-savage-2.3.2-1.el6.x86_64 xorg-x11-drv-siliconmotion-1.7.5-1.el6.x86_64 xorg-x11-drv-sis-0.10.3-1.el6.x86_64 xorg-x11-drv-sisusb-0.9.4-1.el6.x86_64 xorg-x11-drv-synaptics-1.4.1-3.el6.x86_64 xorg-x11-drv-tdfx-1.4.3-2.el6.x86_64 xorg-x11-drv-trident-1.3.4-1.el6.x86_64 xorg-x11-drv-v4l-0.2.0-4.el6.x86_64 xorg-x11-drv-vesa-2.3.0-2.el6.x86_64 xorg-x11-drv-vmmouse-12.7.0-1.el6.x86_64 xorg-x11-drv-vmware-11.0.3-1.el6.x86_64 xorg-x11-drv-void-1.4.0-1.el6.x86_64 xorg-x11-drv-voodoo-1.2.4-1.el6.x86_64 xorg-x11-drv-wacom-0.13.0-6.el6.x86_64 xorg-x11-drv-xgi-1.6.0-11.el6.x86_64 xorg-x11-font-utils-7.2-11.el6.x86_64 xorg-x11-server-common-1.10.6-1.el6.centos.x86_64 xorg-x11-server-utils-7.5-5.2.el6.x86_64 xorg-x11-server-Xorg-1.10.6-1.el6.centos.x86_64 xorg-x11-utils-7.4-8.el6.x86_64 xorg-x11-xauth-1.0.2-7.1.el6.x86_64 xorg-x11-xinit-1.0.9-13.el6.x86_64 xorg-x11-xkb-utils-7.4-6.el6.x86_64 [root@dz ~]# Oddly, as you can see above, SSH seems to properly negotiate the X11 Forwarding, yet I still have no DISPLAY. I compared the ssh -vv output between hope and dy to the one shown above between hope and dz and they match very closely, but only the dy destination actually gives me the DISPLAY setting and properly forwards X11. I tried "ssh -Y" rather than "ssh -X" but got the same symptom. I also tried clearing IP tables entirely. What should I try next, oh wizards? -Steve