=head1 OmniPITR - omnipitr-restore =head2 USAGE /some/path/omnipitr/bin/omnipitr-restore [options] %f %p Options: =over =item --data-dir (-D) Where PostgreSQL datadir is located (path) (defaults to current working directory). =item --source (-s) Where I can find wal segments to use. Check L for more details. =item --recovery-delay (-w) Delay when recovering wal segments (in seconds). This is primarily used to keep window of safety before I will be applied on slave database. =item --finish-trigger (-f) Name of file to watch for existence - if it exists, recovery process will stop, and PostgreSQL slave will become fully functional. Check L section for more details. =item --remove-unneeded (-r) [optional name] Makes I remove unneeded wal segments. These are B segments that were passed to Pg - I checks last redo segment to make sure this is safe. This option has optional value being last segment needed in walarchive (i.e. all earlier than this can be removed). This is specifically designed, so you can put: omnipitr-restore ... -r %r ... as your restore_command in recovery.conf. %r works in PostgreSQL since 8.3 version, so if you're using older Pg, you can't use it (though you can still use -r without value, and let omnipitr-restore find out what it can delete on its own). =item --remove-before (-rb) If I<--remove-unneeded> is enabled, this option makes sure that segment removal will happen also before returning segment to PostgreSQL. In case there are many unapplied WAL segments in archive, not having I<--remove-before> means that old, obsolete, files will be removed only after PostgreSQL will catchup with replication. With I<--remove-before> old files will be removed during catching up. =item --remove-at-a-time (-rt) When removing old segments, remove at most that many segments before re-entering loop to check for signals and/or new segment availability for Postgres. Defaults to 3. =item --removal-pause-trigger (-p) Name of file to watch for existence. If it exists - I will not remove unneeded wal segments regardless of I<--removal-unneeded> option. This is to provide a way to make backups on slave. =item --pre-removal-processing (-h) If given, argument will be treated as shell command to run when any segment will be removed from archive. If the hook will finish without errors - segment will be removed. If there will be errors - removal procedure will be postponed, and after some time, it will be retried. There will be one extra parameter attached, which will be name of the segment file to be processed (prepared in such a way that it will be relative to current working directory). Passed segment will always be uncompressed. =item --temp-dir (-t) Where to create temporary files (defaults to /tmp or I<$TMPDIR> environment variable location). This is only used when using pre-removal-processing. =item --log (-l) Name of logfile (actually template, as it supports %% L markers. Unfortunately due to the %x usage by PostgreSQL, We cannot use %% macros directly. Instead - any occurence of ^ character in log dir will be first changed to %, and later on passed to strftime. Please note that on some systems (Solaris for example) default shell treats ^ as special character, which requires you to quote the log filename (if it contains ^ character). So you'd better write it as: --log '/var/log/omnipitr-^Y-^m-^d.log' =item --streaming-replication (-sr) This option should be used if you're setting streaming replication slave. It causes I to die as soon as it will be called for WAL segment that is not in walarchive. =item --pid-file Name of file to use for pidfile. If it is specified, than only one copy of I (with this pidfile) can run at the same time. Trying to run second copy of I will result in an error. =item --verbose (-v) Log verbosely what is happening. =item --gzip-path (-gp) Full path to gzip program - in case you can't set proper PATH environment variable. =item --bzip2-path (-bp) Full path to bzip2 program - in case you can't set proper PATH environment variable. =item --lzma-path (-lp) Full path to lzma program - in case you can't set proper PATH environment variable. =item --lz4-path (-ll) Full path to lz4 program - in case you can't set proper PATH environment variable. =item --xz-path (-xz) Full path to xz program - in case you can't set proper PATH environment variable. =item --pgcontroldata-path (-pp) Full path to pg_controldata program - in case you can't set proper PATH environment variable. =item --error-pgcontroldata (-ep) Sets handler for errors with pgcontroldata. Possible options: =over =item * ignore - warn in logs, but nothing else to be done - after some time, recheck if pg_controldata works =item * hang - enter infinite loop, waiting for admin interaction, but not finishing recovery =item * break - breaks recovery, and returns error status to PostgreSQL (default) =back Please check L section below for more details. =item --version (-V) Prints version of I, and exists. =item --help (-?) Prints this manual, and exists. =item --config-file (--config / --cfg) Loads options from config file. Format of the file is very simple - each line is treated as argument with optional value. Examples: --verbose --host 127.0.0.1 -h=127.0.0.1 --host=127.0.0.1 It is important that you don't need to quote the values - value will always be up to the end of line (trailing spaces will be removed). So if you'd want, for example, to have magic-option set to "/mnt/badly named directory", you'd need to quote it when setting from command line: /some/omnipitr/program --magic-option="/mnt/badly named directory" but not in config: --magic-option=/mnt/badly named directory Empty lines, and comment lines (starting with #) are ignored. =back =head2 DESCRIPTION Call to I should be in I variable in I. Which options should be given depends only on installation, but generally you will need at least: =over =item * --data-dir PostgreSQL "%p" passed file path is relative to I, so it is required to know it. =item * --log to make sure that information is logged someplace about archiving progress =item * --source to specify where to load WAL segments from =back If you'll specify more than 1 destination, you will also need to specify I<--state-dir> Of couse you can provide many --dst-local or many --dst-remote or many mix of these. Generally omnipitr-restore will try to deliver WAL segment to all destinations, and will fail if B of them will not accept new segment. Segments will be transferred to destinations in this order: =over =item 1. All B destinations, in order provided in command line =item 2. All B destinations, in order provided in command line =back In case any destination will fail, I will save state (which destinations it delivered the file to) and return error to PostgreSQL - which will cause PostgrerSQL to call I again for the same WAL segment after some time. State directory will be cleared after every successfull file send, so it should stay small in size (expect 1 file of under 500 bytes). When constructing command line to put in I PostgreSQL GUC, please remember that while providing C<"%p" "%f"> will work, I requires only "%p" =head3 Source specification If the wal segments are compressed you have to prefix source path with compression type followed by '=' sign. Allowed compression types: =over =item * gzip Decompresses with gzip program, used file extension is .gz =item * bzip2 Decompresses with bzip2 program, used file extension is .bz2 =item * lzma Decompresses with lzma program, used file extension is .lzma =item * lz4 Decompresses with lz4 program, used file extension is .lz4 =item * xz Decompresses with xz program, used file extension is .xz =back If you want to pass any extra arguments to compression program, you can either: =over =item * make a wrapper Write a program/script that will be named in the same way your actual compression program is named, but adding some parameters to call =item * use environment variables All of supported compression programs use environment variables: =over =item * gzip - GZIP =item * bzip2 - BZIP2 =item * lzma - XZ_OPT =item * xz - XZ_OPT =back For details - please consult manual to your choosen compression tool. =back =head3 Finishing recovery There are 2 ways I can finish recovery, and there are 2 separate ways to signal it that it should finish. First, the finishing procedures: =over =item * smart In this mode I will feed all available WAL segments to PostgreSQL (without any I<--recovery-delay> induced delay), and then finish restoration process. =item * immediate In this mode I will skip all pending WAL segments, and make PostgreSQL finish recover immediately. This can be useful in case of running really bad query (think: TRUNCATE users), and wanting to prevent this change to be replicated to slave. =back Now. I can be signaled into finishing recovery in 2 ways, one of which is optional. =over =item * trigger file This one is optional. If you will use --finish-trigger switch, I will look for this file, and if it exists - it will start finishing. If the file exists, and contains string "NOW" (without quotation characters, but with optional new line character "\n"), I will enter "immediate finish" procedure. If the content is different, or the file is empty - it will proceed in smart finish mode. After OmniPITR will finish recovery, and PostgreSQL will enter normal mode of working, it's strongly advised to remove this file. =item * system signal This one works always, regardless of --finish-trigger switch. Generally you can send system signals (kill) to I to make it go to finish recovery procedure. Only 1 signals are supported: =over =item * SIGUSR1 makes the finish I =back It is currently not possible to forcing 'smart' finishing by signal, due to the fact that L is restarted after every segment. =back =head3 Segment removal omnipitr-restore will automatically remove segments that are no longer necessary. To make it happen, it will periodically run I program, and check name of last segment required for redo. If pre-removal-processing is defined, it will be called before actuall removal. omnipitr-restore will remove segments chronologically - oldest segments first. One useful idea for pre-removal-processing, is using L for processing - to send xlog segments to some permanent storage place. =head2 ERRORS =head3 pg_controldata Sometimes, for not yet known reason, pg_controldata fails (doesn't print anything, exits with status -1). In such situation, omnipitr-restore died too, with error, but it had one bad consequence: it made PostreSQL assume that it should stop recovery and go directly into standalone mode, and start accepting connections. Because this solution is not so good, there is now switch to change the behaviour in case of such error. It has 3 possible values. =head4 break "Break" means break recovery, and return to PostgreSQL error. This is default behaviour to make it backward compatible. =head4 ignore With this error handling, omnipitr-restore will simply ignore all errors from pg_controldata - i.e. it will simply print information about error to logs, but it will not change anythin - it will still try to work on next wal segments, and after 5 minutes - it will retry to get pg_controldata. This is the most transparent setting, but also kind of dangerous. It means that if there is non-permanent problem leading to pg_controldata failing not 100%, it might simply get overlooked - replication will work just fine. And this can mean that the real source of the error can propagate and do more harm. =head4 hang With "hang" error handling, in case of pg_controldata failure, omnipitr-restore will basically hang - i.e. enter infinite loop, not doing anything. Of course before entering infinite loop, it will log information about the problem. While it might seem like pointless, it has two benefits: =over =item * It will not cause slave server to become standalone =item * It is easily detactable, as long as you're checking size of wal archive directory, or wal replication lag, or any other metric about replication - as replication will be 100% stoppped. =back To recover from hanged recovery, you have to restart PostgreSQL, i.e. run this sequence of commands (or something similar depending on what you're using to start/stop your PostgreSQL): pg_ctl -D /path/to/pgdata -m fast stop pg_ctl -D /path/to/pgdata start Of course when next pg_controldata error will happen - it will hang again. =head2 EXAMPLES =head3 Minimal setup: restore_command='/.../omnipitr-restore -l /var/log/omnipitr/restore.log -s /mnt/wal_restore/ %f %p' =head3 Minimal setup, but with defined finish trigger and recovery delay (5 mintues): restore_command='/.../omnipitr-restore -D /mnt/data/ -l /var/log/omnipitr/restore.log -s /mnt/wal_restore/ -w 300 -f /tmp/finish.trigger %f %p' =head3 Setup as above, but with pause trigger defined for doing backups-on-slave and removing unneeded segments, working on PostgreSQL 8.3 (or newer) and using its "%r" to denote what can be removed: restore_command='/.../omnipitr-restore -D /mnt/data/ -l /var/log/omnipitr/restore.log -s /mnt/wal_restore/ -w 300 -f /tmp/finish.trigger -r %r -p /tmp/pause.trigger %f %p' =head3 Minimal setup, but with backing up segments to remote server: restore_command='/.../omnipitr-restore -l /var/log/omnipitr/restore.log -s /mnt/wal_restore/ -h "/.../omnipitr-archive --force-data-dir -l /var/log/omnipitr/archive.log -dr bzip2=rsync://backup/postgres/xlogs/" %f %p' =head2 COPYRIGHT The OmniPITR project is Copyright (c) 2009-2013 OmniTI. All rights reserved.