Page 1 of 1

Server problems

Posted: 10 Dec 2013, 14:06
by webwit
There was a problem with our server, it had a bad crash which was visible after contacting hetzner (our hoster) and getting access through a KVM console.

On reboot it showed some SSD errors, and some database tables were corrupted. I repaired this, but I don't know if there is damage.

I declare this SSD now unreliable and uptime uncertain.

I'm currently pulling a second full backup from last night to my local machine (we have one at hetzner backup servers too, but just in case). After this is complete, hetzner will replace the SSD. Downtime is currently unknown.

EDIT: There was a second crash the same day. So far no new crash. We will move to a new server with HDD soon.

Posted: 10 Dec 2013, 14:09
by matt3o
darn SSDs!

Posted: 10 Dec 2013, 14:12
by webwit
Maybe we should go back to normal hard disks, it's not like we need SSD for performance. Any opinions?

Posted: 10 Dec 2013, 14:16
by 7bit
I'm not a big fan of SSDs. SSDs are only good for static data, not for frequently updated data. I bet Daniel Beardsmore killed it with his extreme amount of wiki contributions.
:evilgeek:

Posted: 10 Dec 2013, 14:21
by matt3o
on server I would use them just for very volatile data, such as cache, or disposable logs, but definitely not for "storage". The SSD should be something that you replace and no harm is made to the application. So maybe the system could reside on the SSD but I would put (or backup) data on HDD presto!

If you can backup/duplicate the database (and images/media) into an HDD almost in realtime, that would be perfect. Even 5mins delay would be acceptable I think.

PS: this is especially true for extremely inexpensive hosts such as hetzner

Posted: 10 Dec 2013, 14:23
by pasph

Posted: 10 Dec 2013, 14:26
by Daniel Beardsmore
7bit wrote:I bet Daniel Beardsmore killed it with his extreme amount of wiki contributions.
:evilgeek:
Very probably. I have the Touch of Doom, after all.

Posted: 10 Dec 2013, 14:31
by webwit
I don't like it, so I'll move to a server with HDD.

Posted: 10 Dec 2013, 14:46
by matt3o
with all that RAM a good caching system would do so much more than SSD :)

Posted: 10 Dec 2013, 18:36
by Muirium
SSDs are great for system volumes and for personal machines, but a server ought to use memory to cache and have a hard drive based data store and backup system. Glad to see nothing (that I can see) has been lost with today's surgery.

Posted: 10 Dec 2013, 18:41
by mr_a500
I thought Deskthority was getting hacked like in that Geekhack disaster.

Posted: 10 Dec 2013, 18:44
by matt3o
let me know if you need help with the server... I can do some sysadmin ;)

Posted: 10 Dec 2013, 18:57
by webwit
So hetzner offers to replace the SSD, I tell them I plan to move to a HDD but if they have any tips out of experience.
We have quite good experience with our SSDs. It shouldn't be a problem at all if
you use a software raid 1.
Hmm yeah. No problem at all. (we're running raid 1)

Posted: 10 Dec 2013, 19:05
by matt3o
oh wait, are you on RAID1? so you shouldn't have lost any data. raid 1 doesn't prevent failure to happen but if it happens you have double copy of your data.

that being said a 2 drives raid1 is a bit a of nonsense (during sync how do you know which data is corrupted if you have only 2 drives? 50:50)

Posted: 10 Dec 2013, 19:08
by webwit
Still crashed and corrupted mysql tables :(
But raid 1 might have saved us from an unbootable system. I have a feeling this still might happen.

Posted: 10 Dec 2013, 19:15
by matt3o
webwit wrote:Still crashed and corrupted mysql tables :(
But raid 1 might have saved us from an unbootable system. I have a feeling this still might happen.
data should be safe on one of the two drives if it's a hardware issue. you should mount them separately and check. it happened to me once. I removed the damaged drive and data was safe on the other. anyway not easy to handle these cases remotely

Posted: 10 Dec 2013, 19:18
by webwit
That's one of the reasons I just want to move to a new server with the backup and leave the wreckage behind.

Posted: 10 Dec 2013, 19:47
by webwit
Ordered a new server. Same one, but with HDD.

Posted: 10 Dec 2013, 20:11
by kint
just curious, is there/ how much is the difference between a SSD server and a HDD one?

Posted: 10 Dec 2013, 20:15
by webwit
2 x 240 GB SATA 6 Gb/s SSD vs 2 x 2 TB SATA 6 Gb/s 7200 rpm HDD (both Software RAID1). The HDD server is EUR 10 cheaper per month.
I doubt you'll feel a difference in page/load time response.

Posted: 10 Dec 2013, 20:18
by webwit
Ok, they already have the new server installed.

Posted: 10 Dec 2013, 20:20
by Daniel Beardsmore
By "backup" and "wreckage", you're referring to the operating system, as opposed to rolling back all changes made to Deskthority including the forum and wiki?

Posted: 10 Dec 2013, 20:21
by 7bit
Great! 8x more disk space than before!
:o

Let's discuss how we can use it up!
:evilgeek:
Daniel Beardsmore wrote:By "backup" and "wreckage", you're referring to the operating system, as opposed to rolling back all changes made to Deskthority including the forum and wiki?
Well, this is the downside of it.
:cry:

But a fresh start can always be viewed as a new chance to do it right from the beginning!
:mrgreen:

Posted: 10 Dec 2013, 20:22
by kint
webwit wrote:2 x 240 GB SATA 6 Gb/s SSD vs 2 x 2 TB SATA 6 Gb/s 7200 rpm HDD. The HDD server is EUR 10 cheaper per month.
I doubt you'll feel a difference in page/load time response.
my thought, thanks. Not worth spending the money (and taking the risks with SSDs) in this scenario then imo.

Posted: 10 Dec 2013, 20:23
by webwit
I'll make a fresh backup of the forum and wiki just before the move, so no data is lost. I also made one a couple of hours ago. However, if for some reason the system collapses before the move, the backup used will be max 24 hrs old.