Galera Cluster SST problem with xtrabackup

I ran into a problem when one of my Galera nodes had to do a SST state transfer. I really struggled with it because the logs were not indicating at all what the problem was. Finally I got lucky and figured it out. It seems I was missing the tmpdir parameter from my.cnf. Although MySQL, or in this case MariaDB was able to use the system default /tmp, wsrep or xtrabackup was not. There was a /tmp/percona-version-check file that was only writable and readable by root. Deleting that file solved the problem. Here are the cryptic log messages from both the joiner and the donor nodes:

DONOR:

error.log:

WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log (20131211 20:24:52.065)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20131211 20:24:52.068)
131211 20:24:52 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup –role ‘donor’ –address ‘10.99.0.102:4444/xtrabackup_sst’ –auth ‘root:*HIDDEN*’ –socket ‘/var/lib/mysql/mysql.sock’ –datadir ‘/var/lib/mysql/’ –defaults-file ‘/etc/my.cnf’ –gtid ‘3f679640-4d55-11e3-b061-ebb0a2213f88:118489’
131211 20:24:52 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup –role ‘donor’ –address ‘10.99.0.102:4444/xtrabackup_sst’ –auth ‘root:*HIDDEN*’ –socket ‘/var/lib/mysql/mysql.sock’ –datadir ‘/var/lib/mysql/’ –defaults-file ‘/etc/my.cnf’ –gtid ‘3f679640-4d55-11e3-b061-ebb0a2213f88:118489’: 22 (Invalid argument)
131211 20:24:52 [Warning] WSREP: Could not find peer: 838fbf37-6291-11e3-8a45-67aba358260a
131211 20:24:52 [Warning] WSREP: 1 (sql44): State transfer to -1 (left the group) failed: -1 (Operation not permitted)

innobackup.backup.log:

….
innobackupex: Executing a version check against the server…
Can’t use an undefined value as an ARRAY reference at /usr//bin/innobackupex line 1048.

JOINER:

error.log:

131211 20:17:35 [Note] WSREP: Requesting state transfer: success, donor: 0
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 0 2 (20131211 20:17:35.775)
WSREP_SST: [ERROR] Data directory /var/lib/mysql/ needs to be empty for SST: Manual intervention required in that case (20131211 20:17:35.778)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20131211 20:17:35.781)
131211 20:17:35 [Warning] WSREP: 0 (sql5): State transfer to 1 (sql3) failed: -1 (Operation not permitted)
131211 20:17:35 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.