Moderators Popular Post AY Mod Posted March 31, 2022 Moderators Popular Post Share Posted March 31, 2022 Firstly, my apologies for the extended and unplanned absence of the site since Monday 21st March. It was evident that all was not well with site performance for a few weeks. I had been repeatedly advising and questioning the host’s (Dediserve) support since late January. Repeatedly they would push the blame back saying there was nothing wrong there so it must be software/configurations/heavy loads. Eventually on 3rd February they had to restart the drives due to an issue at their London data centre and they also replaced one of the drives; for a couple of weeks after this the performance was better but the issues gradually returned. Repeated contacts were made with the same rebuttals; regular intervention was required to tweak settings and clear caches which would return service for a few hours. Requests for a further restart were refused on the basis it would affect other clients. This reached intolerable levels during a supposed week off. Due to concerns over poor performance and support plans were already in hand and preparations largely completed before that week off. We had held discussions with the president of Invision (the software used for the site) about migrating the site to them as soon as practicable. Finally, on 21st March, everything started collapsing. Requests for information were met with vague responses about investigation. The following day a technical report was sent… Quote At approximately 08:00 GMT on the 21st of March 2022, our monitoring systems alerted us to degraded performance on a single hypervisor in our LON2 location. Our engineers, upon investigation, initially rebooted the server in an attempt to restore normal service. Following the reboot the RAID array showed a single drive had dropped from the array and the controller was attempting to rebuild it back into the array. A second drive in the array indicated that it was predicted to fail. We continued to monitor the rebuild progress throughout the morning. As the rebuild continued to show errors the decision was taken to evacuate VMs at 14:37. This proved impossible given the current state of the disk array so our engineers progressed to replace one of the two failing drives. This replacement was completed but has led to filesystem corruption on the affected virtual machines. The RAID redundancy on this hypervisor can only tolerate one drive failure at a time. The failure of two drives at the same time has resulted in RAID consistency failure and therefore data loss. So much for the successful drive replacement they reported to me on 4th March! At that stage we were looking at a total data loss by Dediserve as the backups were within that array. An earlier backup from around a year ago was found intact. We had taken the precaution of taking the sql database away from Dediserve onto an alternative platform (again due to performance issues) a year ago so we had all the user/post content safely separate so the drive failures related to the uploaded content such as images. The backup file was enormous, in website terms, which entailed logistical problems in transferring this to Invision. So what does this mean? The transfer to Invision means that the hosting will be with the software providers so that performance should be better integrated. Invision’s systems utilise Amazon’s cloud platforms for improved resilience with continual backups. Improved software support. The uploaded images from the last year could, sadly, not be recovered from Dediserve. Older images should be largely intact although further checks are being made. Users will be able to edit earlier posts and re-upload images if they wish. It’s far from ideal and I’m furious with Dediserve for their failings in this regard. It’s evident at this stage that some saved avatars haven’t been ported across so please re-upload up those when you get opportunity. If the navigations bar or other links do not appear to be working can you please clear any RMweb browsing history in your browser. Once again, I apologise for the interruption in service. Thank you for the emails, Facebook posts, phone calls, text and WhatsApp messages etc – although a distraction when trying to get things moving your support was seen and appreciated. Of course there will still some knobbers but I can’t cure everything. Other issues may be identified through the course of the coming days; your patience will be appreciated. 78 144 6 20 111 Link to post Share on other sites More sharing options...
Popular Post woodenhead Posted March 31, 2022 Popular Post Share Posted March 31, 2022 Pity about the loss of photos over the past 12 months, but could have been much much worse. Thank goodness you'd taken the SQL to another host, there have been some doom moungers on other sites predicting the end but because there was a plan to fix the dediserve mess then when the sh*t literally hit the fan there was a route out of the nightmare. To be fair you needed this as much as a slap across the face at the Oscars, but hopefully you'll look back in 2-3 months and it will be a fading memory. Thank you for your dedication And I didn't even have to log back in, my existing cookies were sufficient to remember me, how good is that. 34 20 1 2 Link to post Share on other sites More sharing options...
RMweb Premium Popular Post melmerby Posted March 31, 2022 RMweb Premium Popular Post Share Posted March 31, 2022 Nice to see you back after all the frustrations caused by Dediserve's woeful performance of recent times. TBH migration to a differnt host seemed to be becoming an obvious choice due to the problems. 15 5 1 Link to post Share on other sites More sharing options...
Popular Post KeithHC Posted March 31, 2022 Popular Post Share Posted March 31, 2022 Well done Andy great to see you back. Keith 12 17 1 Link to post Share on other sites More sharing options...
RMweb Gold Popular Post westerhamstation Posted March 31, 2022 RMweb Gold Popular Post Share Posted March 31, 2022 Hi Andy, Just to say thank you to you and all the others involved in restoring RMweb. All the best Adrian. 12 11 2 1 Link to post Share on other sites More sharing options...
Popular Post woodenhead Posted March 31, 2022 Popular Post Share Posted March 31, 2022 I had to start reading Reddit as an alternative to RMWeb, so I am really glad it's back there is only so much talk of the Oscars and fake controversy I can stand. I wonder how many announcements by Accurascale I've missed in the past week 😄 Ooh new emoji style. 5 1 18 Link to post Share on other sites More sharing options...
RMweb Gold goldngreen Posted March 31, 2022 RMweb Gold Share Posted March 31, 2022 Good to see you back. Shame about the images. I have replacements that I can upload. 4 1 1 Link to post Share on other sites More sharing options...
RMweb Gold Popular Post Sweet pea Posted March 31, 2022 RMweb Gold Popular Post Share Posted March 31, 2022 Andy thanks for all the hard work you have done to make this better. 11 12 2 Link to post Share on other sites More sharing options...
RMweb Gold Popular Post Harlequin Posted March 31, 2022 RMweb Gold Popular Post Share Posted March 31, 2022 It's great to be back! Thanks for the work you've done to get RMWeb back online, @AY Mod, and thanks for the explanation. A Data Centre that backs up data into the same RAID array as the source and fails to spot when that RAID array is failing, is incompetent! I hope you are seeking compensation for the service downtime and lost data! P.S. What's a "knobber"??? 😉 8 24 1 Link to post Share on other sites More sharing options...
RMweb Gold Popular Post Revolution Ben Posted March 31, 2022 RMweb Gold Popular Post Share Posted March 31, 2022 Hi Andy Thank for all your hard work in getting the site back. We've missed it! cheers Ben A. 7 13 1 Link to post Share on other sites More sharing options...
PenrithBeacon Posted March 31, 2022 Share Posted March 31, 2022 🤗 Link to post Share on other sites More sharing options...
RMweb Premium curlypaws Posted March 31, 2022 RMweb Premium Share Posted March 31, 2022 (edited) Great to see RMweb back - it was much missed. And thanks for all your work Andy. It sounds like a very fraught time. Edited March 31, 2022 by curlypaws Fixed the Rmweb capitalisation! 2 4 1 Link to post Share on other sites More sharing options...
RMweb Gold teaky Posted March 31, 2022 RMweb Gold Share Posted March 31, 2022 Glad RMweb is back. Well done Andy! 🎖️ 5 1 1 Link to post Share on other sites More sharing options...
MyRule1 Posted March 31, 2022 Share Posted March 31, 2022 I must be lucky in that in about 35 years of managing large IT systems I never had to deal with a outage such as this, as others have said it's mainly due to, what seems to be, extremely poor infrastructure management by Dediserve. An early IT manager of mine always looked at the worst outcomes and refused to have our backups in the same city so, and this was BR, sent tapes to Crewe from London a couple of times a week! The last datacentre application I managed was sold to us that although our main RAID arrays were in London, they were backed up in Leeds. From Andy's account of events at least he held the SQL etc..to enable this transfer, Well done at the RMweb team for dealing with this as they did. Oddly the advert at the bottom of my first look at the site today was from IONIS saying "Need web hosting for your next project? Choose a host with industry leading uptime." 6 4 1 1 Link to post Share on other sites More sharing options...
s182ggu Posted March 31, 2022 Share Posted March 31, 2022 Great Job, Andy - Thanks. Now you can maybe take your week off!! 7 2 1 Link to post Share on other sites More sharing options...
BernardTPM Posted March 31, 2022 Share Posted March 31, 2022 Well done to you and all your team in the face of adversity. 3 5 1 Link to post Share on other sites More sharing options...
RMweb Gold Popular Post RedgateModels Posted March 31, 2022 RMweb Gold Popular Post Share Posted March 31, 2022 1 hour ago, AY Mod said: At that stage we were looking at a total data loss by Dediserve as the backups were within that array. Who puts a backup on the same RAID Array? Good riddance to Dediserve, add them to the blacklist that's been built up over the years 6 15 1 Link to post Share on other sites More sharing options...
Mike 84C Posted March 31, 2022 Share Posted March 31, 2022 Andy, thanks for all your hard work getting RM back up and running. And the explanation of the problems, me I'd just want to go and get a big gun and resolve the b------t. 😎 Mick 5 Link to post Share on other sites More sharing options...
MarkC Posted March 31, 2022 Share Posted March 31, 2022 Andy & team - congratulations on resolving what must have been an utter nightmare. Thanks again - & well done 👍 Cheers Mark 2 8 1 Link to post Share on other sites More sharing options...
RMweb Gold Derails Models Posted March 31, 2022 RMweb Gold Share Posted March 31, 2022 To add to the comments already, glad it's all back up and running! I know the pain data loss can cause first hand, so to be able to recover to this position is fantastic! Thank-you for your hard work Andy! 7 1 Link to post Share on other sites More sharing options...
RMweb Gold SouthernRegionSteam Posted March 31, 2022 RMweb Gold Share Posted March 31, 2022 Andy, something tells me that you wished you had taken your holiday now rather than a couple of weeks ago...(!). Sod the beer fund, you need a holiday fund! In all seriousness, it's great to have RMweb back, and I dread to think how much stress this whole situation has caused you; and from many directions (all of which entirely preventable and/or unnecessary). 'Thanks' doesn't feel enough considering the dedication you have put into RMweb over the years, even whilst you're supposed to be on holiday. Your efforts are much appreciated! The fact that anything was salvaged is impressive, and seems solely to be down to your foresight. Hopefully this is the start of a more stable and stress-free (OK, less stress - let's be realistic!) RMweb for you and the team. 13 4 1 Link to post Share on other sites More sharing options...
RMweb Gold adb968008 Posted March 31, 2022 RMweb Gold Share Posted March 31, 2022 Wow… just wow. thanks for all your hard work there Andy ! 5 6 1 Link to post Share on other sites More sharing options...
RMweb Gold Andy 53B Posted March 31, 2022 RMweb Gold Share Posted March 31, 2022 Hooray!!! No more cold turkey. Thanks very much Andy and the rest of the team, here's to a brighter future 2 5 1 Link to post Share on other sites More sharing options...
RMweb Premium Hilux5972 Posted March 31, 2022 RMweb Premium Share Posted March 31, 2022 Great to see the site back Andy. Thanks for your hard work and dedication. 2 6 1 Link to post Share on other sites More sharing options...
John Tomlinson Posted March 31, 2022 Share Posted March 31, 2022 Well done and thank you for all your hard work! John. 1 2 1 Link to post Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now