I feel like pages have got slower to load here. Not sure if it’s my connection. Anyone else finding the same?
Disk I/O looks high. (top shows a high value for cpu waiting, and iotop reports a unicorn process with a value of 90%+ in the IO column (sometimes 99%!)) Don’t have a good frame of reference, unfortunately; will try restarting some things…
EDIT: Ran a cleanup job. Thought it helped, but then it struggled to save this edit for this comment. Hrm.
Also seeing 502s sometimes.
Did some more hunting around. Looks like the problem may well be a slow disk (well, virtual disk). Since we haven’t had problems in the past, would guess the hosting provider is having some trouble and it’s rippling out, rather than this being a hard limit. So it may fix itself. Doubt there’s anything we can do to fix things. Will keep looking around, though.
Seems worse when logged in. Would guess this is because more is written to disk because the software tries to keep track of what people have read and things like that. Hosting provider reported troubles on March 3rd, so could be related.
(yeah, login itself often errors out. Curious if all users are seeing the same.)
Going to try rebuilding tomorrow. The site took an unexpectedly long time to build with the last update, and that update coincided with some problems on the provider, so wondering if those might have had a knock-on effect.
Have also tried deleting all tags, since tags have caused headaches on other things in the past.
Restarted the box. If that doesn’t help, will do rebuild tomorrow, as planned.
It’s slower for me too. I thought it was my connection, but other sites seem to be loading normally.
And just got several 502s trying to post this comment.
Yeah, it’s annoying. I can see that it’s timing out server-side and I can see roughly what’s happening, but not WHY. There are ‘unicorn processes’ to handle requests to the server and when one of those takes longer than 30 secs, it gets killed, and that’s when we see the errors. Those processes seem to be heavy on IO (not sure if that indicates a problem or if that’s expected, different docs say different things -_-). So might be able to stop the errors by increasing the timeout on those processes, but that doesn’t answer why things are slow in the first place, and things will still be slow. SQL and redis look fine so it doesn’t look like a problem with the DB or the message queue; I think it has to be either a bug in the forum software, a bug in the software it depends on to run, or an issue with our hosting provider (rather than: eg: some configuration we need to change). We don’t seem to be being hammered by bots, either; there’s a normal amount of bot activity. Going to do the rebuild shortly.
Rebuild done. It seems to have helped… But am wary! Things have seemed to have helped before!
It’s zippy on this account, now. What a hassle.
Login was quick today!
1 Like