Several months ago, I’d switched to using Percona’s xtrabackup & innobackupex for all of my mysql backup needs. I had successfully used these backups to restore and replicate databases across several systems. It is good stuff.
Last week, I needed to set up new replication of an 80gb database. This should have been routine by now, but when I attempted to prepare the backup this time, it whined and complained and failed. I was kind of frazzled by the time I gave up on the issue and declared it a fluke of one sort or another.
Last night, I tried again from Sunday’s full backup, and it happened again:
ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=. xtrabackup Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64) xtrabackup: cd to . xtrabackup: This target seems to be not prepared yet. xtrabackup: xtrabackup_logfile detected: size=75546624, start_lsn=(86 1293090752) xtrabackup: Temporary instance for recovery is set as followings. xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:512M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 75546624 xtrabackup: Starting InnoDB instance for recovery. xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752 090818 2:54:34 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... 090818 2:54:34 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. InnoDB: File name .//tmp/#sql6e1e_8cce1_0.ibd InnoDB: File operation call: 'create'. InnoDB: Cannot continue operation.
I gave up after poking a few things.
This morning’s fresh look turned up this bug report.
ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo mkdir tmp ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=. xtrabackup Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64) xtrabackup: cd to . xtrabackup: This target seems to be not prepared yet. 090818 12:27:41 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find. xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry. xtrabackup: xtrabackup_logfile detected: size=84983808, start_lsn=(86 1293090752) xtrabackup: Temporary instance for recovery is set as followings. xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:512M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 84983808 xtrabackup: Starting InnoDB instance for recovery. xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752 090818 12:27:41 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Doing recovery: scanned up to log sequence number 86 1298333184 (6 %) InnoDB: Doing recovery: scanned up to log sequence number 86 1303576064 (13 %) InnoDB: Doing recovery: scanned up to log sequence number 86 1308818944 (20 %) InnoDB: Doing recovery: scanned up to log sequence number 86 1314061824 (27 %) InnoDB: Doing recovery: scanned up to log sequence number 86 1319304704 (34 %) 090818 12:27:44 InnoDB: Starting an apply batch of log records to the database... ... snip ...
That’s right. There’s a bug in innodb restoration that interprets location of /tmp (configurable in my.cnf) to be relative in stead of absolute.
So, if you have problems while trying to restore from an xtrabackup/ibbackup snapshot (or if you’re trying to recover innodb after a crash), just creating the offending tmp directory appears to work.