VEX Forum Shut Down (was: Looks like Tony tripped over the ethernet cable)

I’m primarily a VEX person, so this is my first CD post…

So to those who aren’t yet aware, VEX Forum has been (very likely) taken down by IFI. If it isn’t a coincidence it is in response to this post (if you haven’t read it please do).


The timing is very suspicious, and gives us some early insight into what IFI’s public response is going to be to one of VEX’s prominent public figures being exposed like this.

Though CD and VF share an administrator, and people commonly associate with CD being managed by IFI, I wouldn’t worry about it being taken down. VEX Forum is hosted on DataBank, a hosting company that hosts most of IFI’s, its subsidiaries’, infrastructure. By extension VEX Forum is likely under IFI’s full control. On the contrary, chiefdelphi is hosted on DigitalOcean, a company known for hosting smaller sites (personal projects, communities and the like, though some larger companies also use their services). So CD is likely completely under the control of Brandon’s personal DigitalOcean account, who is unlikely to take it down.

Hope this helped clear some things up

- m

(PS: I created an archive of VF content a few days ago in the case of this happening, it’s currently only in a machine-readable format but if anyone wants it they could download it here [~5 GB])

41 Likes

I was under the impression CD was hosted on Linode. I wonder if they changed providers recently.

1 Like

Your thread title reminded me of this xkcd comic:

46 Likes

Here is the info on chiefdelphi.com. It does appear to have some connection to linode (in particular, it uses Linode for DNS services), but the main IP address (the A record) points to an IP which is owned by DigitalOcean.

However, DigitalOcean being the host doesn’t insulate CD at all from admin action by anyone with admin access (which, as has been pointed out, includes IFI’s head of IT).

9 Likes

Looks like all that archiving paid off! It really is sad that they aren’t dealing with this responsibly.

12 Likes

I’d be careful to speculate publicly. Yes, there are lots of reasons folks who could’ve taken the VEX forums down may have wanted to. Yes, the timing is suspicious. Yes, coincidences happen.

I’m not saying you’re wrong – you’re probably right – but we should at least give folks a day to see if we hear anything.

Don’t put the pitchforks back in the shed, but maybe put them down and ready to redeploy for a few minutes.

26 Likes

Right. Timing is suspicious but sites go down for a variety of reasons . Give it some time to find out.

2 Likes

If the plug was pulled, then shame on vex.

If it truly is an accident or something unrelated, then hopefully they say so cause it looks terrible on them.

9 Likes

I think it was taken offline intentionally. In fact, the hosting server itself is still online.

How do I figure?

Well, for starters, you can ping it:

But, you may ask, “that’s just pinging the IP address! Maybe that’s just pinging some databank.com IP address while the actual server fell off the rack!”

Except…the server hasn’t fallen off its rack. Why? It’s got an SSH port open, just like your RoboRIO. You can test this by opening a connection with a random username.

And that SSH server responds just fine. You wouldn’t even get the public key if it was down.

Edit: This is not conclusive evidence. Websites can fail in many extremely convoluted ways, many of which may leave a responsive SSH server but a dead frontend. Also doesn’t help it happened close to 5pm Central Time the week of Thanksgiving.


That said, this, plus the fact that connections to the server got suddenly cut right after today’s major post, rather than slowly degraded, makes me lean heavily towards this conclusion.

62 Likes

Right. The IP endpoint is still operational but it no longer listens for web requests on port 80. This indicates the Discourse backend application was stopped.

4 Likes

This reeks of cowardice.

31 Likes

Seeing the same behavior here. I concur with Guinea’s analysis, pending more information.

18 Likes

I agree this is suspicious, but not sure your conclusions make sense, the IP could have simply been reassigned to another server with an open 22 port and they failed to update the DNS record.

1 Like

Since I develop and support websites for a living, I’ll chime in to say that there is a fair amount that can go wrong that would take down the server process while leaving other things, like SSH, still up and accessible. For example, just last week we had a case where a database crash took down a site, though literally everything else on the server was still working. That took about an hour to diagnose and fix one morning. In other cases, a number of our deployments aren’t set up by the hosts to auto-start correctly, so whenever they patch the server and restart it, they have to remember to manually start the application server - everything else is up and running just fine by that point! One case a while back, a server blew its motherboard and they spent the better part of a week getting something jerry-rigged so we could migrate their data to another server. And these are mission-critical deployments, not a public user forum.

A number of years ago at a different company, we had a hard drive fail on a server - that took the better part of an entire day to recover from.

All that is to say, wait and see. The timing is certainly suspicious. But if it is innocent, then the timing (after normal work hours) is a pain for anyone responsible for getting it back up and running for them. If it’s not back up by thanksgiving, then I would start to draw conclusions.

19 Likes

No, it’s not. This is simply your browser giving you a cached version from earlier.

5 Likes

Oh, my fault I forgot to clear the cache

3 Likes

48hrs is quite the SLO! Wish I got to respond that late.

I’m a Site Reliability Engineer, so this is the sort of thing I do for work every day - determine if things are down, get them online, and figure out why they’re down (order of these tasks may vary). Jon pretty much hits the nail on the head - there’s more things involved here than malice that may have brought the site down. I am surprised though that IFI allows 22 external through their infrastructure, rather than blocking that port externally / whitelisting IPs from the office / only permitting devices on the network.

While it’s entirely possible the webserver fell over, I have my doubts.

7 Likes

Port 80 is used for web servers–so that would indicate the web server itself (nginx, etc.) is down wouldn’t it? Were the Discourse back end down, you would get a 502 Bad Gateway response from the webserver itself

2 Likes

There’s always the possibility of a hack/ddos/etc.

I haven’t bothered to run Nmap (and IFI seems to be in a litigious mood), but with 22 open, I wonder what other open ports are open on that server with potentially unmaintained software.

With the recent IFI news, I’m sure there’s a few angry skids out there that are either too foolish to realize the potential consequences or just smart enough to try to hide under the veil of being a minor.

TBH, I think this is more likely a deliberate move from someone at IFI/Vex/etc.

4 Likes

That’s gonna depend on the exact infrastructure they are running it on. At the very least, the public facing web server is shut down, or a firewall is blocking ports 80 and 443.

2 Likes