[{"data":1,"prerenderedAt":958},["ShallowReactive",2],{"article-20210901-gitlab-postgresql-recovery":3},{"article":4,"surround":947},{"id":5,"title":6,"authorKey":7,"body":8,"codeLinks":911,"createdAt":912,"description":913,"excerpt":914,"extension":934,"image":935,"meta":936,"navigation":139,"path":937,"published":938,"readingTime":156,"revisedAt":911,"seo":939,"stem":940,"tags":941,"__hash__":946},"articles\u002Farticles\u002F20210901-gitlab-postgresql-recovery.md","GitLab PostgreSQL Data Recovery","philip",{"type":9,"value":10,"toc":904},"minimark",[11,34,49,60,70,73,92,97,100,172,175,254,257,263,266,272,283,410,421,424,453,457,462,501,505,508,583,608,616,659,671,749,762,765,815,834,840,873,876,897,900],[12,13,14,15,22,23,28,29,33],"p",{},"Today, shit happened on a larger on-premise ",[16,17,21],"a",{"href":18,"rel":19},"https:\u002F\u002Fabout.gitlab.com\u002F",[20],"nofollow","GitLab EE"," instance of one of our ",[16,24,27],{"href":25,"rel":26},"https:\u002F\u002Fwww.onlime.ch\u002F",[20],"Onlime GmbH"," customers. GitLab's production.log started to fill up with ",[30,31,32],"code",{},"PG::Error (FATAL: the database system is in recovery mode)"," errors which were somehow related to LFS operations. That definitely didn't sound cool and smelled like data corruption. The customer noticed it by failed CI jobs with 500 Internal Server Errors, and let me know immediately.",[12,35,36,37,42,43,48],{},"As we have that GitLab server running in a LXC container on a ",[16,38,41],{"href":39,"rel":40},"https:\u002F\u002Fopenzfs.org",[20],"ZFS"," based system (",[16,44,47],{"href":45,"rel":46},"https:\u002F\u002Fproxmox.com\u002Fen\u002Fproxmox-ve",[20],"Proxmox VE","), it was easy to pull a clone of the full system and play around with PostgreSQL data recovery before working on live data. I decided to go for a full data restore by dumping and loading it from scratch in a freshly initialized PostgreSQL data dir.",[12,50,51,52,55,56,59],{},"So these were the ",[30,53,54],{},"FATAL"," errors I found in ",[30,57,58],{},"\u002Fvar\u002Flog\u002Fgitlab\u002Fgitlab-rails\u002Fproduction.log"," (this is a Omnibus install of GitLab EE):",[61,62,67],"pre",{"className":63,"code":65,"language":66},[64],"language-text","Processing by Repositories::LfsStorageController#download as HTML\n  Parameters: {\"repository_path\"=>\"group\u002Fproject.git\", \"oid\"=>\"38387ae38...\"}\nPG::Error (FATAL:  the database system is in recovery mode\n):\n\nStarted GET \"\u002Fgroup\u002Fproject.git\u002Fgitlab-lfs\u002Fobjects\u002F372ea928b2...\" for 192.168.x.x at 2021-09-01 09:19:31 +0200\nActiveRecord::ConnectionNotEstablished (FATAL:  the database system is in recovery mode\n):\n","text",[30,68,65],{"__ignoreMap":69},"",[12,71,72],{},"The errors were always preceded by some LFS operation, so that looks correlated.",[74,75,76,83,89],"blockquote",{},[12,77,78,82],{},[79,80,81],"strong",{},"UPDATE 2021-09-01 23:50"," – In the meantime (after the full recovery below - Shame on me ignorant sysadmin!), I have figured out this was actually caused by an OOM issue of that LXC container, where random PostgreSQL processes were killed by the system's oom-killer...",[61,84,87],{"className":85,"code":86,"language":66},[64],"# \u002Fvar\u002Flog\u002Fgitlab\u002Fpostgresql\u002Fcurrent\n2021-09-01_07:19:31.05398 LOG:  server process (PID 10068) was terminated by signal 9: Killed\n\n# \u002Fvar\u002Flog\u002Fsyslog\nSep  1 09:19:31 git kernel: [5068062.072130]  oom_kill_process.cold.33+0xb\u002F0x10\n",[30,88,86],{"__ignoreMap":69},[12,90,91],{},"But I still keep this article to explain how to do a full PostgreSQL data recovery, even though we never had any data corruption. :)",[93,94,96],"h2",{"id":95},"check-postgresql-tools","Check PostgreSQL Tools",[12,98,99],{},"First, check your PostgreSQL version and figure out where GitLab Omnibus install put all those commands:",[61,101,105],{"className":102,"code":103,"language":104,"meta":69,"style":69},"language-bash shiki shiki-themes github-light-default github-light-default github-dark-default","$ gitlab-psql --version\npsql (PostgreSQL) 12.6\n\n$ gitlab-psql -c 'select version()'\nPostgreSQL 12.6 on x86_64-pc-linux-gnu, ...\n","bash",[30,106,107,124,134,141,154],{"__ignoreMap":69},[108,109,112,116,120],"span",{"class":110,"line":111},"line",1,[108,113,115],{"class":114},"sQJ0q","$",[108,117,119],{"class":118},"s-BGO"," gitlab-psql",[108,121,123],{"class":122},"sDkKp"," --version\n",[108,125,127,130],{"class":110,"line":126},2,[108,128,129],{"class":114},"psql",[108,131,133],{"class":132},"s_bOF"," (PostgreSQL) 12.6\n",[108,135,137],{"class":110,"line":136},3,[108,138,140],{"emptyLinePlaceholder":139},true,"\n",[108,142,144,146,148,151],{"class":110,"line":143},4,[108,145,115],{"class":114},[108,147,119],{"class":118},[108,149,150],{"class":122}," -c",[108,152,153],{"class":118}," 'select version()'\n",[108,155,157,160,163,166,169],{"class":110,"line":156},5,[108,158,159],{"class":114},"PostgreSQL",[108,161,162],{"class":122}," 12.6",[108,164,165],{"class":118}," on",[108,167,168],{"class":118}," x86_64-pc-linux-gnu,",[108,170,171],{"class":118}," ...\n",[12,173,174],{},"Check database sizes:",[61,176,178],{"className":102,"code":177,"language":104,"meta":69,"style":69},"$ gitlab-psql -c 'SELECT pg_database.datname as \"dbname\", pg_database_size(pg_database.datname)\u002F1024\u002F1024 AS size_in_mb FROM pg_database ORDER by size_in_mb DESC'\n       dbname        | size_in_mb \n---------------------+------------\n gitlabhq_production |      29151\n template0           |          7\n template1           |          7\n postgres            |          7\n",[30,179,180,191,206,211,222,233,243],{"__ignoreMap":69},[108,181,182,184,186,188],{"class":110,"line":111},[108,183,115],{"class":114},[108,185,119],{"class":118},[108,187,150],{"class":122},[108,189,190],{"class":118}," 'SELECT pg_database.datname as \"dbname\", pg_database_size(pg_database.datname)\u002F1024\u002F1024 AS size_in_mb FROM pg_database ORDER by size_in_mb DESC'\n",[108,192,193,196,200,203],{"class":110,"line":126},[108,194,195],{"class":114},"       dbname",[108,197,199],{"class":198},"s5nWI","        |",[108,201,202],{"class":114}," size_in_mb",[108,204,205],{"class":132}," \n",[108,207,208],{"class":110,"line":136},[108,209,210],{"class":114},"---------------------+------------\n",[108,212,213,216,219],{"class":110,"line":143},[108,214,215],{"class":114}," gitlabhq_production",[108,217,218],{"class":198}," |",[108,220,221],{"class":114},"      29151\n",[108,223,224,227,230],{"class":110,"line":156},[108,225,226],{"class":114}," template0",[108,228,229],{"class":198},"           |",[108,231,232],{"class":114},"          7\n",[108,234,236,239,241],{"class":110,"line":235},6,[108,237,238],{"class":114}," template1",[108,240,229],{"class":198},[108,242,232],{"class":114},[108,244,246,249,252],{"class":110,"line":245},7,[108,247,248],{"class":114}," postgres",[108,250,251],{"class":198},"            |",[108,253,232],{"class":114},[12,255,256],{},"You'll find the following symlink:",[61,258,261],{"className":259,"code":260,"language":66},[64],"\u002Fusr\u002Fbin\u002Fgitlab-psql -> \u002Fopt\u002Fgitlab\u002Fbin\u002Fgitlab-psql\n",[30,262,260],{"__ignoreMap":69},[12,264,265],{},"The other commands are here:",[61,267,270],{"className":268,"code":269,"language":66},[64],"\u002Fopt\u002Fgitlab\u002Fembedded\u002Fbin\u002Fpg_dump\n\u002Fopt\u002Fgitlab\u002Fembedded\u002Fbin\u002Fpg_dumpall\n",[30,271,269],{"__ignoreMap":69},[12,273,274,275,278,279,282],{},"But DON'T try to do a dump as ",[30,276,277],{},"root",", always do it under user ",[30,280,281],{},"gitlab-psql",":",[61,284,286],{"className":102,"code":285,"language":104,"meta":69,"style":69},"$ su - gitlab-psql\n\n(gitlab-psql)$ echo $PWD           \n\u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql\n(gitlab-psql)$ which pg_dumpall\n\u002Fopt\u002Fgitlab\u002Fembedded\u002Fbin\u002Fpg_dumpall\n\n# connect over socket\n(gitlab-psql)$ pg_dumpall -h \u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql\n# or connect over port\n(gitlab-psql)$ pg_dumpall -h localhost -p 9187\n",[30,287,288,301,305,323,328,344,349,353,360,380,386],{"__ignoreMap":69},[108,289,290,292,295,298],{"class":110,"line":111},[108,291,115],{"class":114},[108,293,294],{"class":118}," su",[108,296,297],{"class":118}," -",[108,299,300],{"class":118}," gitlab-psql\n",[108,302,303],{"class":110,"line":126},[108,304,140],{"emptyLinePlaceholder":139},[108,306,307,310,312,315,317,320],{"class":110,"line":136},[108,308,309],{"class":132},"(",[108,311,281],{"class":114},[108,313,314],{"class":132},")",[108,316,115],{"class":114},[108,318,319],{"class":118}," echo",[108,321,322],{"class":132}," $PWD           \n",[108,324,325],{"class":110,"line":143},[108,326,327],{"class":114},"\u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql\n",[108,329,330,332,334,336,338,341],{"class":110,"line":156},[108,331,309],{"class":132},[108,333,281],{"class":114},[108,335,314],{"class":132},[108,337,115],{"class":114},[108,339,340],{"class":118}," which",[108,342,343],{"class":118}," pg_dumpall\n",[108,345,346],{"class":110,"line":235},[108,347,348],{"class":114},"\u002Fopt\u002Fgitlab\u002Fembedded\u002Fbin\u002Fpg_dumpall\n",[108,350,351],{"class":110,"line":245},[108,352,140],{"emptyLinePlaceholder":139},[108,354,356],{"class":110,"line":355},8,[108,357,359],{"class":358},"s9BHP","# connect over socket\n",[108,361,363,365,367,369,371,374,377],{"class":110,"line":362},9,[108,364,309],{"class":132},[108,366,281],{"class":114},[108,368,314],{"class":132},[108,370,115],{"class":114},[108,372,373],{"class":118}," pg_dumpall",[108,375,376],{"class":122}," -h",[108,378,379],{"class":118}," \u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql\n",[108,381,383],{"class":110,"line":382},10,[108,384,385],{"class":358},"# or connect over port\n",[108,387,389,391,393,395,397,399,401,404,407],{"class":110,"line":388},11,[108,390,309],{"class":132},[108,392,281],{"class":114},[108,394,314],{"class":132},[108,396,115],{"class":114},[108,398,373],{"class":118},[108,400,376],{"class":122},[108,402,403],{"class":118}," localhost",[108,405,406],{"class":122}," -p",[108,408,409],{"class":122}," 9187\n",[12,411,412,413,416,417,420],{},"If you're unsure about the port your PostgreSQL is running, check ",[30,414,415],{},"netstat -tulpen"," (don't need to visit the Netherlands for that!) and grab it from there. If you don't find any running postgres process that binds to some port, try to connect to the socket directly with ",[30,418,419],{},"-h \u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql",".",[12,422,423],{},"For below recovery, consult the following PostgreSQL 12 documentation:",[425,426,427,437,446],"ul",{},[428,429,430],"li",{},[16,431,434],{"href":432,"rel":433},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002F12\u002Fapp-pg-dumpall.html",[20],[30,435,436],{},"pg_dumpall",[428,438,439],{},[16,440,443],{"href":441,"rel":442},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002F12\u002Fapp-initdb.html",[20],[30,444,445],{},"initdb",[428,447,448],{},[16,449,452],{"href":450,"rel":451},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002F12\u002Fbackup-dump.html",[20],"PostgreSQL Backup and Restore",[93,454,456],{"id":455},"data-recovery-migration","Data Recovery Migration",[458,459,461],"h3",{"id":460},"the-idea","The Idea",[463,464,465,468,476,479,482,490,493,498],"ol",{},[428,466,467],{},"Stop all GitLab services, only keep PostgreSQL running",[428,469,470,471],{},"Make a full data dump with ",[16,472,474],{"href":432,"rel":473},[20],[30,475,436],{},[428,477,478],{},"Stop PostgreSQL",[428,480,481],{},"Move current PostgreSQL data dir away, so we can start from scratch",[428,483,484,485],{},"Re-initialize PostgreSQL data dir with ",[16,486,488],{"href":441,"rel":487},[20],[30,489,445],{},[428,491,492],{},"Start PostgreSQL",[428,494,495,496],{},"Restore full data dump by feeding it to ",[30,497,281],{},[428,499,500],{},"Stop all GitLab services, reconfigure GitLab and restart all services",[458,502,504],{"id":503},"dump-restore","Dump \u002F Restore",[12,506,507],{},"Stop all GitLab services and only keep PostgreSQL running:",[61,509,511],{"className":102,"code":510,"language":104,"meta":69,"style":69},"$ gitlab-ctl stop\n$ gitlab-ctl start postgresql\n$ gitlab-ctl status\n\n# You might also want to stop cron, so it won't interfere your migration\n$ systemctl stop cron && systemctl disable cron\n# Don't forget to start and re-enable it again after migration!\n",[30,512,513,523,535,544,548,553,578],{"__ignoreMap":69},[108,514,515,517,520],{"class":110,"line":111},[108,516,115],{"class":114},[108,518,519],{"class":118}," gitlab-ctl",[108,521,522],{"class":118}," stop\n",[108,524,525,527,529,532],{"class":110,"line":126},[108,526,115],{"class":114},[108,528,519],{"class":118},[108,530,531],{"class":118}," start",[108,533,534],{"class":118}," postgresql\n",[108,536,537,539,541],{"class":110,"line":136},[108,538,115],{"class":114},[108,540,519],{"class":118},[108,542,543],{"class":118}," status\n",[108,545,546],{"class":110,"line":143},[108,547,140],{"emptyLinePlaceholder":139},[108,549,550],{"class":110,"line":156},[108,551,552],{"class":358},"# You might also want to stop cron, so it won't interfere your migration\n",[108,554,555,557,560,563,566,569,572,575],{"class":110,"line":235},[108,556,115],{"class":114},[108,558,559],{"class":118}," systemctl",[108,561,562],{"class":118}," stop",[108,564,565],{"class":118}," cron",[108,567,568],{"class":132}," && ",[108,570,571],{"class":114},"systemctl",[108,573,574],{"class":118}," disable",[108,576,577],{"class":118}," cron\n",[108,579,580],{"class":110,"line":245},[108,581,582],{"class":358},"# Don't forget to start and re-enable it again after migration!\n",[74,584,585,591],{},[12,586,587,590],{},[79,588,589],{},"To repeat"," (I found this quite confusing as PostgreSQL does not always bind to a port):",[12,592,593,594,597,598,600,601,603,604,607],{},"After starting ",[30,595,596],{},"postgresql",", always use ",[30,599,415],{}," to check on which port it is running. If it's not running on a port, connect to the socket with ",[30,602,419],{}," instead of ",[30,605,606],{},"-h localhost -p \u003CPORT>",", see examples below.",[12,609,610,611,282],{},"Now, make full gzipped dump with ",[16,612,614],{"href":432,"rel":613},[20],[30,615,436],{},[61,617,619],{"className":102,"code":618,"language":104,"meta":69,"style":69},"$ su - gitlab-psql\n(gitlab-psql)$ pg_dumpall -h \u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql | gzip > pg_dumpall.sql.gz\n",[30,620,621,631],{"__ignoreMap":69},[108,622,623,625,627,629],{"class":110,"line":111},[108,624,115],{"class":114},[108,626,294],{"class":118},[108,628,297],{"class":118},[108,630,300],{"class":118},[108,632,633,635,637,639,641,643,645,648,650,653,656],{"class":110,"line":126},[108,634,309],{"class":132},[108,636,281],{"class":114},[108,638,314],{"class":132},[108,640,115],{"class":114},[108,642,373],{"class":118},[108,644,376],{"class":122},[108,646,647],{"class":118}," \u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql",[108,649,218],{"class":198},[108,651,652],{"class":114}," gzip",[108,654,655],{"class":198}," >",[108,657,658],{"class":118}," pg_dumpall.sql.gz\n",[12,660,661,662,665,666,282],{},"Re-create PostgreSQL data dir ",[30,663,664],{},"\u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql\u002Fdata"," from scratch with ",[16,667,669],{"href":441,"rel":668},[20],[30,670,445],{},[61,672,674],{"className":102,"code":673,"language":104,"meta":69,"style":69},"$ gitlab-ctl stop postgresql\n\n$ su - gitlab-psql\n(gitlab-psql)$ mv data data.12.6.BKUP\n(gitlab-psql)$ initdb data\n\n$ gitlab-ctl start postgresql\n",[30,675,676,686,690,700,719,735,739],{"__ignoreMap":69},[108,677,678,680,682,684],{"class":110,"line":111},[108,679,115],{"class":114},[108,681,519],{"class":118},[108,683,562],{"class":118},[108,685,534],{"class":118},[108,687,688],{"class":110,"line":126},[108,689,140],{"emptyLinePlaceholder":139},[108,691,692,694,696,698],{"class":110,"line":136},[108,693,115],{"class":114},[108,695,294],{"class":118},[108,697,297],{"class":118},[108,699,300],{"class":118},[108,701,702,704,706,708,710,713,716],{"class":110,"line":143},[108,703,309],{"class":132},[108,705,281],{"class":114},[108,707,314],{"class":132},[108,709,115],{"class":114},[108,711,712],{"class":118}," mv",[108,714,715],{"class":118}," data",[108,717,718],{"class":118}," data.12.6.BKUP\n",[108,720,721,723,725,727,729,732],{"class":110,"line":156},[108,722,309],{"class":132},[108,724,281],{"class":114},[108,726,314],{"class":132},[108,728,115],{"class":114},[108,730,731],{"class":118}," initdb",[108,733,734],{"class":118}," data\n",[108,736,737],{"class":110,"line":235},[108,738,140],{"emptyLinePlaceholder":139},[108,740,741,743,745,747],{"class":110,"line":245},[108,742,115],{"class":114},[108,744,519],{"class":118},[108,746,531],{"class":118},[108,748,534],{"class":118},[12,750,309,751,754,755,757,758,761],{},[30,752,753],{},"$PWD"," of ",[30,756,281],{}," user is already ",[30,759,760],{},"\u002Fvar\u002Fopt\u002Fgitlab\u002Fpostgresql",", so you don't need to use full pathes in that context)",[12,763,764],{},"Restore dump:",[61,766,768],{"className":102,"code":767,"language":104,"meta":69,"style":69},"$ su - gitlab-psql\n(gitlab-psql)$ zcat pg_dumpall.sql.gz | gitlab-psql -h localhost -p 5432 -d postgres\n",[30,769,770,780],{"__ignoreMap":69},[108,771,772,774,776,778],{"class":110,"line":111},[108,773,115],{"class":114},[108,775,294],{"class":118},[108,777,297],{"class":118},[108,779,300],{"class":118},[108,781,782,784,786,788,790,793,796,798,800,802,804,806,809,812],{"class":110,"line":126},[108,783,309],{"class":132},[108,785,281],{"class":114},[108,787,314],{"class":132},[108,789,115],{"class":114},[108,791,792],{"class":118}," zcat",[108,794,795],{"class":118}," pg_dumpall.sql.gz",[108,797,218],{"class":198},[108,799,119],{"class":114},[108,801,376],{"class":122},[108,803,403],{"class":118},[108,805,406],{"class":122},[108,807,808],{"class":122}," 5432",[108,810,811],{"class":122}," -d",[108,813,814],{"class":118}," postgres\n",[74,816,817,822],{},[12,818,819],{},[79,820,821],{},"NOTE:",[12,823,824,829,830,833],{},[16,825,828],{"href":826,"rel":827},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002F12\u002Fbackup-dump.html#BACKUP-DUMP-ALL",[20],"SQL Dump: Using pg_dumpall",": Actually, you can specify any existing database name to start from, but if you are loading into an empty cluster then ",[30,831,832],{},"postgres"," should usually be used.",[12,835,836,837,839],{},"Stop ",[30,838,596],{}," and reconfigure GitLab, then start all GitLab services:",[61,841,843],{"className":102,"code":842,"language":104,"meta":69,"style":69},"$ gitlab-ctl stop postgresql\n$ gitlab-ctl reconfigure\n$ gitlab-ctl start\n",[30,844,845,855,864],{"__ignoreMap":69},[108,846,847,849,851,853],{"class":110,"line":111},[108,848,115],{"class":114},[108,850,519],{"class":118},[108,852,562],{"class":118},[108,854,534],{"class":118},[108,856,857,859,861],{"class":110,"line":126},[108,858,115],{"class":114},[108,860,519],{"class":118},[108,862,863],{"class":118}," reconfigure\n",[108,865,866,868,870],{"class":110,"line":136},[108,867,115],{"class":114},[108,869,519],{"class":118},[108,871,872],{"class":118}," start\n",[12,874,875],{},"Monitor logs:",[61,877,879],{"className":102,"code":878,"language":104,"meta":69,"style":69},"$ tail -f \u002Fvar\u002Flog\u002Fgitlab\u002Fgitlab-rails\u002Fproduction.log \u002Fvar\u002Flog\u002Fgitlab\u002Fpostgresql\u002Fcurrent\n",[30,880,881],{"__ignoreMap":69},[108,882,883,885,888,891,894],{"class":110,"line":111},[108,884,115],{"class":114},[108,886,887],{"class":118}," tail",[108,889,890],{"class":122}," -f",[108,892,893],{"class":118}," \u002Fvar\u002Flog\u002Fgitlab\u002Fgitlab-rails\u002Fproduction.log",[108,895,896],{"class":118}," \u002Fvar\u002Flog\u002Fgitlab\u002Fpostgresql\u002Fcurrent\n",[12,898,899],{},"All good!",[901,902,903],"style",{},"html pre.shiki code .sQJ0q, html code.shiki .sQJ0q{--shiki-light:#953800;--shiki-default:#953800;--shiki-dark:#FFA657}html pre.shiki code .s-BGO, html code.shiki .s-BGO{--shiki-light:#0A3069;--shiki-default:#0A3069;--shiki-dark:#A5D6FF}html pre.shiki code .sDkKp, html code.shiki .sDkKp{--shiki-light:#0550AE;--shiki-default:#0550AE;--shiki-dark:#79C0FF}html pre.shiki code .s_bOF, html code.shiki .s_bOF{--shiki-light:#1F2328;--shiki-default:#1F2328;--shiki-dark:#E6EDF3}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s5nWI, html code.shiki .s5nWI{--shiki-light:#CF222E;--shiki-default:#CF222E;--shiki-dark:#FF7B72}html pre.shiki code .s9BHP, html code.shiki .s9BHP{--shiki-light:#6E7781;--shiki-default:#6E7781;--shiki-dark:#8B949E}",{"title":69,"searchDepth":126,"depth":136,"links":905},[906,907],{"id":95,"depth":126,"text":96},{"id":455,"depth":126,"text":456,"children":908},[909,910],{"id":460,"depth":136,"text":461},{"id":503,"depth":136,"text":504},null,"2021-09-01","Recovering PostgreSQL data for GitLab by a full dump\u002Frestore.",{"type":9,"value":915},[916,926],[12,917,14,918,22,921,28,924,33],{},[16,919,21],{"href":18,"rel":920},[20],[16,922,27],{"href":25,"rel":923},[20],[30,925,32],{},[12,927,36,928,42,931,48],{},[16,929,41],{"href":39,"rel":930},[20],[16,932,47],{"href":45,"rel":933},[20],"md","\u002Fimages\u002Fcovers\u002Fgitlab-meets-postgresql.png",{},"\u002Farticles\u002F20210901-gitlab-postgresql-recovery",false,{"title":6,"description":913},"articles\u002F20210901-gitlab-postgresql-recovery",[942,943,944,945,596],"sysadmin","linux","backup","gitlab","8p9YmorJejeueDPrH9NpCmcycKeGC1QWitHgoUiOrzg",[948,953],{"title":949,"path":950,"stem":951,"createdAt":952,"children":-1},"Integrate Algolia InstantSearch into a Vue Project","\u002Farticles\u002F20210830-algolia-vue-instantsearch","articles\u002F20210830-algolia-vue-instantsearch","2021-08-30",{"title":954,"path":955,"stem":956,"createdAt":957,"children":-1},"Fail2ban jail to mitigate DoS attacks against Apache","\u002Farticles\u002F20210915-fail2ban-apache-dos","articles\u002F20210915-fail2ban-apache-dos","2021-09-15",1794961390516]