notes on a disrupted Saturday morning

David_Snowball

Dear friends,

MFO was taken off-line by our hosting company for a several hours on Saturday morning. They concluded that we'd generated extraordinary server load over the past couple days. We got a warning note on 7:45 a.m., suggesting that we review our data demands and consider a dedicated server. Ninety minutes later we were taken offline because we threatened the stability of the server which housed us. Chip, later joined by James, one of her IT staff colleagues, and Accipiter, worked to resolve the challenge and reason with the server folks. By early afternoon we were back online.

The question of server load is a bit tough for us to analyze because it's not necessarily related to the traffic we draw or the number of pages you view. One bad bit of computer code might generate a huge demand on a server by asking it to endlessly repeat the same action (think about Abbott and Costello's old "who's on first" routine). Chip and Accipiter reviewed our database files and disabled a number of plug-ins, especially those related to the board software. We've requested heightened monitoring from our hosting company - both so that we could create a baseline for normal server demand but also as an early warning system for demand spikes. We haven't yet heard back from them on whether they'll cooperate.

What's next? Chip and James, her colleague, have signed us up for a Cloud Flare account - a cloud-based service which has the prospect of noticeably improving our speed though not directly addressing the problem of downtime, have created a backup copy of MFO and will soon be able to test other server options. Together those actions create the possibility for us to migrate to a dedicated and better-monitored server. It will take some time and will incur some additional operating expense, but we think both are well worth it if the result is a more stable site that's always available to you.

So, regrets for any inconvenience and thanks to Chip, James and Accipiter for making a massive effort on no notice, on a day they'd normally have off.

As ever,

David

catch22

Good Gosh ! No inconvenience.

A most large thank you to everyone involved. I would buy the drink and dinner right now if possible.

David, you noted: "We haven't yet heard back from them on whether they'll cooperate."

A long time associate and a most convincing fella I know, Mr. Bozo Morelli; could stop at the facility and have a "chat", as he is only 15 minutes distance from Troy, Mi. He "works" in the general region of the site.

Thank you again for everyones time and efforts.

Respectfully,

Catch

Zolta

I use the board daily. I also get tons of mail asking me for $3, 5, 10, 25, 50, 100, or $250 "donations" to this or that political party or newsblogs or PAC. I'd much rather contribute to MFO. Please consider a (small) monthly charge. It'd probably make you know your usage requirements better. I'm getting guilt cramps fearing that I'm not reading enough books from Amazon.com.
You're one of the few WWW activities that would be worth a subscription.

MaxBialystock

David, bless you for all your work, AND the whole crew: Chip, Accipiter, James, et al.

AndyJ

Many thanks to the emergency response crew.

bnath001

thanks for everyone.

Anna

Thanks from me too.

Old_Skeet

I am not sure if this will help ... but, at the time I was trying to edit one of my recent comments to one of Ted's post when I received an error message and then there was nothing. Hey, if I took the board down I am extremely sorry ... but, I don't think so. My thinking is that there were perhaps too many trying to access this post, all at the same time, while it was under, or being taken to, edit mode. Again, I hope this information helps.

Thanks for all you do.

Skeeter

David_Snowball

Reply to @Skeeter: Yes, yes. I've long lived in dread of the prospect that your army of Twitter followers would discover that "Skeeter" lays behind their beloved idol, and that a mad rush squealing followers would descend in pursuit of you.

Such is life on the web.

Cheers,

David

kallerid

use the AMZN link for MFO guys to get a few shekels so this great site can be free and not a burden.

BTW, Can you ask the RPHYX folks how the new fund could be UP 0.01 on a 14 basis point increase in 10yr? Is it less rate sensitive than usual fund?

David_Snowball

Reply to @kallerid: Sherman has about 40% of the RSIVX portfolio in the same securities as in the very conservative RPHYX, and was planning on letting cash accumulate a bit. In general, he imagines the fund living somewhere between the durations of the short-term and intermediate-term groups.

Here's a suggestion: write RiverPark and ask. You'd be surprised at how willing smaller firms are to communicate.

As ever,

David

Old_Skeet

Reply to @David_Snowball:

Hey, I don't Tweet so there must be some misinformation and/or you have me consfused with someone else.

Thanks again, for all you do.

Skeeter

David_Snowball

Reply to @Skeeter: (it was humor!)

Old_Skeet

Reply to @David_Snowball:

Perhaps … But, I took the site going down very seriously as a fatal error message appeared on my screen as the site went down. It listed a bunch of addresses so it was either something I might have done, although I don’t think so, or the board got crashed form too many trying to access Ted’s post while it was under perhaps edit mode, by me, in attempting to edit a comment I had previously made. In any event … I think it is something worth exploring in more depth. If you want me to try and replicate what I did just let me know and if it crashes again ... we will then know the root of the problem.

And remember , you heard it first here from Olde Skeet! (Just some humor)

David_Snowball

Reply to @Skeeter: For what interest it holds, we now think it was our security system. Every site has a front end, the part you and I see, and a back end, the part that administrators see. The back end of our site controls our control panel and our ties to the server. Bots regularly (hundreds to thousands of times an hour) try to break in to the back end. Every time they do, our security program logs their IP address (a series of nine numbers identifying a computer) and permanently bans, or "blacklists" them.

Our security log grew to two megabytes and every time someone tried to access the back end of the site, the software had to check the incoming IP address against every one in the blacklist. We guessing that a thousand or more run-throughs an hour of the blacklist might have been what pushed us over the edge.

We can't tell for sure but we're watching and we've got other plans in process.

Keep a good thought for us!

David