Skyscraper City Forum banner
1 - 20 of 27 Posts

·
Banned
Joined
·
5,921 Posts
Discussion Starter · #1 ·
What the hell is going on with the website ? It keeps going down every few minutes. Me thinks a new website host is required.
 

·
Registered
Joined
·
4,747 Posts
They have an incompetent sysadmin, it's as simple as that.

Is it a coincidence that one of the most unreliable websites on the internet happens to have a sysadmin who tweets his thanks to the people DDoSing the site for helping him to learn things? I think not. These are things he should have known about and mitigated long ago, not things he should be winging when some script kiddie points a couple of machines at him. Every fairly well known website encounters routine DDoS attacks and the like, but how many of them have so many periods of downtime whilst the sysadmins tweet about being 'on it'?

Any other website would have kicked him to the kerb long ago.
 

·
Registered
Joined
·
4,747 Posts
Lots. You can't mitigate against DDoS attacks, not a site with SSC's resources, anyway.
Yes you can, in fact it's quite easy to mitigate the majority of threats on a very small budget. It isn't remotely normal for a site to experience these levels of downtime. It's gross incompetence and DaiTengu should have been shown the door long ago.
 

·
10th February 2008
Joined
·
60,582 Posts
Unfortunately this has been going on for quite a while now, and I don't mean just today.

Sort it out please. :)
 

·
Registered
Joined
·
16,736 Posts
They have an incompetent sysadmin, it's as simple as that.

Is it a coincidence that one of the most unreliable websites on the internet happens to have a sysadmin who tweets his thanks to the people DDoSing the site for helping him to learn things? I think not. These are things he should have known about and mitigated long ago, not things he should be winging when some script kiddie points a couple of machines at him. Every fairly well known website encounters routine DDoS attacks and the like, but how many of them have so many periods of downtime whilst the sysadmins tweet about being 'on it'?

Any other website would have kicked him to the kerb long ago.
This is an incredibly disrespectful attitude, especially as he has been working his arse off to sort it out (who do you think set us up on cloudfare to try and avoid this?).

We are one of the largest discussion forums on the net (yes, we are) that is entirely dependent on ad-revenue to support itself, we don't have the cash to chuck on anything more robust.
 

·
Registered
Joined
·
4,747 Posts
This is an incredibly disrespectful attitude, especially as he has been working his arse off to sort it out (who do you think set us up on cloudfare to try and avoid this?).
No it isn't disrespectful at all, he is completely incompetent and should fall on his sword for the sake of the community. I could work for days on end at a loom but it doesn't make me a weaver, it makes me someone playing about with a loom. If I didn't have the talent and ability to use that loom I wouldn't be offering my services as a weaver to people. If I claimed to be one and then spent hours to create a ragged patch of random bits of cotton I would expect people to call me out on my incompetence.

I'm fairly sure there things in life he is good at, but being a sysadmin most definitely isn't one of them.

We are one of the largest discussion forums on the net (yes, we are) that is entirely dependent on ad-revenue to support itself, we don't have the cash to chuck on anything more robust.
Basic threat mitigation costs very little money. The fact that it's taken this long to even think of going down the Cloudflare route speaks volumes about his incompetence, never mind the fact that most threats can be neutralised with no such outsourcing.

Do you really think, regardless of budget, that a well configured system with sufficient resources for it's day to day operation should suffer the levels of downtime it does? We aren't talking about one off short downtime here, we're talking frequent and extended. It simply would not and should not happen with a well configured and well maintained setup.
 

·
Registered
Joined
·
4,747 Posts
^ yes problems are always very easy and cheap to fix when they're not yours, cheers.
The majority are actually free to fix Jan, not just cheap, as the tools are open source and require no additional hardware. The fact of the matter is that you have someone completely clueless performing what is a fairly technical role for you.

You generally find in life that people who don't know what they are doing make something appear to be very expensive and suggest outsourcing important aspects of the work to others. It's not because these things actually are that expensive or difficult, it's because they don't know how to do it themselves. If they really were that expensive or difficult then everyone would suffer from the similar levels of downtime.

Do you honestly believe that it is normal or acceptable for a site to suffer these levels of downtime? It's amazing how even something as simple as implementing Cloudflare functionality was royally screwed up. It's absolute gross incompetence from someone who really has no business calling himself a sysadmin.
 

·
Registered
Joined
·
4,747 Posts
I think you should give Irwell the job, he sounds like he'd be communicative and level headed in a crisis.
I'd say I'm being perfectly level headed. It's not like this is a one-off incident, the site is regularly down for extended periods. The 'crisis', as with all the previous ones, is entirely of DaiTengu's making. If a soldier kept shooting his colleagues instead of the enemy would you expect them to keep giving him more bullets and saying "try again"?

I'm not personally particularly bothered that the site goes down so frequently, it's not like I post on here all that regularly anyway, I'm just shocked that this level of incompetence seems to be so readily accepted by the people who run the site.

Surely a line has to be drawn in the sand at some point as regards all these downtime incidents and someone has to say that enough is enough?
 

·
Moderator
Joined
·
35,227 Posts
Yes you can, in fact it's quite easy to mitigate the majority of threats on a very small budget. It isn't remotely normal for a site to experience these levels of downtime. It's gross incompetence and DaiTengu should have been shown the door long ago.
which we have now done successfully... before this we had NO ddos protection at all, which did rather surprise me when i found out (this was undoubtedly an oversight). cloudflare worked for a while but now we have other solutions.

and if you want to know why it takes so long to get it done? because the sysadmin has to get permission to spend money.
 

·
Registered
Joined
·
4,747 Posts
which we have now done successfully... before this we had NO ddos protection at all, which did rather surprise me when i found out (this was undoubtedly an oversight). cloudflare worked for a while but now we have other solutions.
That is beyond an oversight if true. DDoS protection is a living thing, with constantly evolving threats that require you to tweak your configuration. Even with outsourced protection you still need to maintain your own security mechanisms on your own network for those cases where the attack is directly against your network itself.

To not have such protection in place would mean not knowing you had to perform these very fundamental tasks, much like the only reason to not turn the steering wheel when driving down a winding road would be because you didn't know how to drive. It's highly unlikely that someone who knew how to drive would keep on heading towards a wall and not think about steering, in the same way that a competent sysadmin would not sit there as new exploits and techniques became known without looking to mitigate them. It's a core element of the job.

and if you want to know why it takes so long to get it done? because the sysadmin has to get permission to spend money.
I doubt there was a requirement to spend any money at all. Considering the number of things your current sysadmin overlooks you'll probably find hundreds of changes that could be made to reduce the impact of DDoS attacks. For example, properly configured firewalls can screen out a large percentage of this traffic, as you appear to have learnt this time around, without the need for additional spend.

Once you'd agreed to spend some money on Cloudflare, however, you then had substantial downtime beyond that point because of configuration issues between Cloudflare and your network. It's a 30 minute job at most to set up to route traffic through Cloudflare even for a complex network, plus maybe a little longer for the DNS changes to propagate. Overlooking something so basic as firewall rules leading to it dropping packets from Cloudflare is shocking.

I'm being serious here when I say you should drop him like a stone before someone who really knows what they are doing hits you rather than some kids with a script. It's very dangerous letting someone play sysadmin with the data, servers and bandwidth at your disposal as it could do serious harm in the wrong hands.

On another note, you don't appear to have acted on the e-Privacy Directive. You probably should act on that one as well considering you are based in the EU and are targeting EU users.
 

·
Moderator
Joined
·
35,227 Posts
I doubt there was a requirement to spend any money at all. Considering the number of things your current sysadmin overlooks you'll probably find hundreds of changes that could be made to reduce the impact of DDoS attacks. For example, properly configured firewalls can screen out a large percentage of this traffic, as you appear to have learnt this time around, without the need for additional spend.
i can't tell you how wrong you are about what has been done. i'm not even going to break down what has been done here as this is a public forum, but i can assure you it has nothing to do with firewall rules. the DDOS was so large that at one point all the other websites that our webhosting company has were knocked out as collateral damage. it was massive. we get them all the time, but this was on a completely different scale. whoever it was had access to many thousands and thousands of computers. such a value of a hacked network is an enormous amount of money and it was directed at this site.
 

·
Registered
Joined
·
4,747 Posts
i can't tell you how wrong you are about what has been done. i'm not even going to break down what has been done here as this is a public forum, but i can assure you it has nothing to do with firewall rules.
Funny that, because on the other thread your sysadmin said pretty much precisely that.

Yeah, we got hit with a DDoS attack by the same kids who hit us about 6 weeks ago (at the end of May). Right now it's more of an issue tweaking our DDoS filter, as I believe it's seeing CloudFlare's IPs as a potential attack vector, hence why they're being blocked.
Also, I note that you now have firewall rules that block IPs that perform multiple syn scans which you didn't previously have.

the DDOS was so large that at one point all the other websites that our webhosting company has were knocked out as collateral damage. it was massive. we get them all the time, but this was on a completely different scale. whoever it was had access to many thousands and thousands of computers. such a value of a hacked network is an enormous amount of money and it was directed at this site.
You should already have had a setup that detected spikes in traffic and either dropped packets, amended routing tables or updated your DNS records to reroute traffic through services like Cloudflare automatically depending on traffic levels and the type of exploitation being used. You weren't even blocking syn scans until this week.
 

·
Moderator
Joined
·
35,227 Posts
just stop arguing. you clearly know very little about our precise situation. why the hell would i, or mike, or anyone else say what we have done. this is a public forum.

it's not our fault, and certainly not mike's if the scale of the DDOS is so great, it smashes the hosting company to pieces and causes their entire network to fall over because the upstream provider is overwhelmed. that's how large it peaked at.
 

·
Registered
Joined
·
4,747 Posts
just stop arguing. you clearly know very little about our precise situation. why the hell would i, or mike, or anyone else say what we have done. this is a public forum.

it's not our fault, and certainly not mike's if the scale of the DDOS is so great, it smashes the hosting company to pieces and causes their entire network to fall over. that's how large it peaked at.
The scale of the attack was probably so great precisely because you didn't employ any decent security techniques. The majority of DDoS attacks lead to exponentially increased traffic when your systems are responding to the packets rather than simply dropping them, which yours were not doing.

Even now, despite putting some intrusion prevention in place like detecting syn scans, you still have unnecessary ports open on a global basis. Why leave ports like SSH, FTP and NFS accessible to anyone anywhere and risk exploitation of bugs or vulnerabilities? If nothing else, at least put IP filters in place for these ports. We're talking sysadmin 101 here, not anything complicated.
 
1 - 20 of 27 Posts
Top