As we hope users have noticed, the recent performance issues on the AO3 have been alleviated! \0/ The fix came courtesy of our lovely Systems team, who tweaked our server settings to add more unicorns! Unicorns are the workers which actually make the requests of the servers - if there are too many, then things slow down because the servers are overloaded, but if there are too few, then things slow down because we are refusing requests we could potentially serve. (It's a whole unicorn balancing act!)
We weren't optimistic about how much difference changing our unicorn settings would make, because we know we have some underlying application issues which can bog down the database. So, we're really pleased that Sidra was able to find a new balance for the unicorns which has produced noticeable improvements in performance.
Meanwhile, behind the scenes, many of our other fine staff have been working on the performance issues. Sysadmin James_ has been working on partitioning the database: data is stored in tables, and in the new version of MySQL we're using, there's a way to split these tables into parts without affecting anything else. This means that you can search within a much smaller table rather than the whole big block of data, which makes everything faster. With the help of Systems staff, our coders have been going through the slow query database, looking for places where the code runs slowly and needs rewriting to be more efficient. James_ has also been looking into caching options for us - we'd like to use Squid or something similar (the unicorns need some tentacly friends), but we have a lot of different options which affect exactly what is displayed on the page, so we need to figure out the best ways of dealing with those.
Senior coder Elz is also continuing her work on rewriting our browsing filters, which are one of the biggest problem areas in terms of performance - she hopes to have the first version of this ready for testing soon.
We're really happy that the unicorns have helped us address the immediate issue, but we know this doesn't get us off the hook - we still have lots of work to do! While much of this work goes on behind the scenes, we wanted our users to have a glimpse of what's happening when we say 'we're working on it'. :)
We'd like to say a big thank you to all the coders and sysadmins who've been working hard on dealing with these issues. We'd also like to say thank you to all the users who have sent us messages of support and encouragement - we really appreciate it and it's a big boost when we're struggling with some tricky issues. Thanks for your patience while we continue to improve the site. <3