AO3 News

Release Notes 0.8.19

Published: 2012-06-13 07:46:29 -0400

Yet another update from your tireless archive volunteers! James from our Systems Committee has been making adjustments behind the scenes to stabilize the servers and get the most out of our caching, and we've seen some good improvements there. At the same time, we've been working on improving or scaling back the areas of our code where changes will give us the biggest gains.

Filtering

In order to improve performance further, tag filtering on work listing pages is disabled for the time being, until we roll out our new system. You can read more about this change in our post on disabling filters. We know this is an inconvenience for many users, but the filters are really the 800-pound gorilla sitting on top of our database - the works pages are both the most popular and the slowest on the site, which is a bad combination. We've had plans to fix them for a while, and that's underway. However, we need a few more weeks to finish and deploy the upgrade, since it also affects our search engine and quite a lot of our code. Our top priority is to make sure works remain accessible to users, and that new works and feedback can be posted and accessed. Looking carefully at our code and our stats, we concluded that removing filtering was the best way to ensure these goals in the short-term.

You'll still be able to view all the works for a particular tag, view the works for a user or collection in a particular fandom, and use our search feature to refine your results. Our post on disabling filters includes some handy tips to help you find what you're looking for. We hope to have full functionality restored to you soon! As a bonus side effect of this change, we've been able to remove the 1000 work limit on lists of works. This is because without the filters we can rely on the pagination system to limit the amount that we retrieve from the database at one time. So, while you can't filter your results any more, you CAN go through and read every work posted in your fandom! We hope this will compensate a little for the inconvenience.

Work Stats Caching

We've also done more caching of work stats (all the counts of comments, bookmarks, hits, etc.), so you may notice that these update more slowly on index pages now. The information is all still being recorded; we're just waiting a little longer to go get the counts for each work to spread out the load.

People Listings

The alphabetical people listings on the People page weren't actually that useful for finding users, and they were another performance drain.

We've replaced the full alphabetical listing with a listing of 10 random users, and added emphasis on the search. Note you can use wildcards in the search, so if you're not sure of someone's name you can enter part of it followed by an asterisk to get similar names. For example, entering Steve* would get Steve_Rogers_lover, SteveMcGarrettsGirl, stevecarrellrocks, etc.

Invitation Requests

We've suspended user requests for additional invitations for now as well. If you need invitations urgently for a challenge or for an archive rescue project, please contact Support. We also fixed an issue that potentially allowed users to snoop for other emails in the waiting queue.

Thank you!

Thanks to everyone who has been working hard on these issues, especially James, who has put in lots of hours tweaking the servers, and Elz, who has been doing the heavy lifting on code changes. Thanks also to all of you for your patience and understanding while we work!

And finally...

The great news is that so far, this emergency measure does seem to be having a noticeable effect. Our server load has diminished dramatically since we deployed this change:

Graph showing server load, with a mark showing the time of the deploy. The load drops dramatically from this time onwards.

Comment

Disabling filters: information and search tips

Published: 2012-06-12 19:58:16 -0400

Key information: As an emergency measure to deal with recent performance issues, we have disabled browsing filters on the site (the grey box of choices which appears on work index pages). This is a temporary measure to ensure that as many people as possible can access the site. You can still use our tags and advanced search feature to find the works you want. As an additional bonus, removing the filters has allowed us to remove the 1000 works cap on lists of works, so you can browse through all the works in your fandom! Read on for more information!

What's happening

As detailed in our recent post on performance, our coders and sys-admins are continuing to work on the performance issues we've been experiencing. We've made some server adjustments which have alleviated some of the worst problems, but we still need to make some substantial changes to fix the issues. We're aware that lots of users are still unable to access the site; as an emergency measure, we've decided to disable tag filters, which put a very heavy load on our servers. This means that the grey box with tags you can check to filter a list of works will no longer appear on the work index pages. We know this will be an inconvenience for many users, but the filters are really the 800-pound gorilla sitting on top of our database. Removing them for now will mean that people can access the site, even if they can't browse quite as easily as usual.

We've been working on significantly redesigning the part of our code that handles filtering for a while - because it's a major performance burden on some of the most popular pages of the site, refactoring this code to make it more efficient has been a priority for some time now. We're almost done with the rewritten version, but it needs more work and extended testing before we roll it out. (We want to be sure it doesn't introduce new bugs.) So, the filters will go away for a few weeks, and will then be replaced by the new, rewritten version.

One major disadvantage of the way the filters were designed was that they needed to retrieve the tags from the list of works found in order to build the filter options. This meant that we had to limit the number of works returned at one time to 1000, because otherwise building the filters would take too long. A side bonus of removing the filters is that we've been able to remove the 1000 works cap! The browsing redesign in progressaims to work around this issue, so we hope to avoid re-introducing this limitation when filtering returns.

How can I find the works I want?

Although the removal of the filters will make it harder to browse the works listings for specific things, there are still lots of ways to find the works you need.

Fandoms page

If you're looking for a specific fandom, you can browse the Fandoms page. Fandoms are organised by media type; the easiest way to find a particular fandom is to use Ctrl + F (or Command + F on a Mac) to search the page in your browser. The fandom pages will give you a list of all the works in your fandom; unfortunately there will be no way to filter that list down further.

Tags

Clicking on any tag will still bring up works with that tag, or with any tag marked as a synonym. So, if you click on Riza Hawkeye you'll get all the works tagged with 'Riza Hawkeye', 'Riza', 'Riza is awesome', etc. Again, while the filters are disabled there'll be no way to filter this list further.

Advanced Search

If you want more refined control over which works you find, you will need to use our Work Search. This feature could use a little bit of prettifying, but the underlying search is quite powerful. Use the following tips to help you find exactly the works you want:

  • A space equals AND. So, entering Fluff Friendship would find you works tagged with both 'fluff' and 'friendship'
  • | equals OR. So, entering Homestuck | My Little Pony will find you works tagged with 'Homestuck' AND/OR 'My Little Pony'
  • - equals NOT. So, entering Supernatural - Castiel/Dean Winchester will find works tagged Supernatural, but will exclude those tagged Castiel/Dean Winchester.
  • Fandom, Character, Relationship, Rating, Category, and Warning are all classed as tags (as well as the 'Additional tags'). So, you can search for works which are Explicit, or exclude works tagged 'Major Character Death'.
  • Using quotes around a phrase will search for that exact phrase. So, "Harry Potter" will get works tagged with 'Harry Potter', whereas Harry Potter will get works tagged with 'Harry' and works tagged with 'Potter'.
  • Entering a term in the tag field will only find works with exactly that tag - so searching for Charles/Erik will bring up only the few works tagged with exactly that tag, not the ones tagged 'Erik Lehnsherr/Charles Xavier' (whereas if you click on the 'Charles/Erik' tag you'll get works with all variations of that pairing).
  • The search has trouble with tags which have dashes in them. If you search for X-Men, for instance, you noticed you'll get lots with X and no X-Men. To get around this, put the tag in quotes: "X-Men".

As well as searching tags, titles, and authors, you can also search for specific word counts, hits, kudos, and dates - including ranges, which is a useful tool for finding fics in a fandom. For example, you can search for all Stargate Atlantis fics published 5-6 years ago.

Some search examples!

  • Find an explicit Fullmetal Alchemist work with the pairing Riza Hawkeye/Roy Mustang, with no Archive Warnings: Enter "Fullmetal Alchemist" "Riza Hawkeye/Roy Mustang" "No Archive Warnings Apply" Explicit.
  • Find works with Rodney McKay but without John Sheppard: Enter "Rodney McKay" -"John Sheppard".
  • Find works tagged with "Alternate Universe" in either the Homestuck or White Collar fandoms: Enter "Alternate Universe" Homestuck | "White Collar".
  • Find all explicit works tagged as angst, but excluding M/M pairings: Enter Angst Explicit -"M/M"

Search bookmarklets

If you find yourself re-using the same search parameters (only T-rated works, only works under 5,000 words, only works with over 10 kudos) for new fandoms or characters you fall in love with, you could give these custom search bookmarklets a try. They are not official AO3 tools, but made by one of our own and utilizing the Advanced Search functionality. Think of them as a saved search that lets you enter a keyword (such as a fandom name or specific kink) and spits out only the kind of work you want to see. For help in putting together your own saved search, don't hesitate to comment on the post or here.

What next?

This is definitely a short term measure, but we think it will have a big effect on site performance. In a few weeks we hope to deploy our all new search and browse features, which will restore more browsing functionality without placing the same load on the servers. We thank you for your patience and understanding while we continue to work on the problem areas.

Post edited 2012-06-13, 12.00 UTC to reflect some minor changes in functionality & bring it up to date.

Comment

Update on AO3 performance issues

Published: 2012-06-11 08:12:17 -0400

Since last month, we've been experiencing frequent and worsening performance problems on the Archive of Our Own as the site has expanded suddenly and dramatically. The number of new users joining the site doubled between April and May, and we currently have over 17,000 users waiting for an invitation. We've been working hard to deal with the 502 errors and site slowdowns, and we've implemented a number of emergency fixes which have slightly alleviated the issues, but these haven't been as effective as we'd hoped. We're confident that we will be able to fix the problems, but unfortunately we expect the next round of fixes to take at least two weeks to implement.

We know that it's really frustrating for users when the site is inaccessible, and we're sorry that we're not able to fix the problems more quickly. We wanted to give you an update on what's going on and what we're doing to fix it: see below for some more details on the problems. While we work on these issues, you should get better performance (and alleviate the load on the servers) by browsing logged-out where possible (more details below).

Why so many problems?

As we mentioned in our previous post on performance issues, the biggest reason for the site slowdowns is that site usage has increased dramatically! We've almost doubled our traffic since January, and since the beginning of May the pace of expansion has accelerated rapidly. In the last month, more than 8,000 new user accounts were created, and more than 31,000 new works were posted. This is a massive increase: April saw just 4,000 new users and 19,000 new works. In addition to the growing number of registered users, we know we've had a LOT more people visiting the site: between 10 May and 9 June we had over 3,498.622 GB of traffic. In the past week, there were over 12.2 million page views - this number only includes the ones where the page loaded successfully, so it represents a lot of site usage!

This sudden and dramatic expansion has come about largely as a result of changes on Fanfiction.net, who have recently introduced more stringent enforcement of their policies relating to explicit fanworks which have resulted in some fans no longer being able to host their works there. One of the primary reasons the AO3 was created was in order to provide a home for fanworks which were at risk of deletion elsewhere, so we're very keen to welcome these new users, but in the short term this does present us with some challenges!

We'd already been preparing for site expansion and identifying areas of the site which needed work in order to ensure that we could grow. This means some important performance work has been ongoing; however, we weren't expecting quite such a rapid increase, so we've had to implement some changes on an emergency basis. This has sometimes meant a few additional unexpected problems: we're sorry if you ran into bugs while our maintenance was in progress.

What we've done so far

Our sys-admins and coders have implemented a number of things designed to reduce the load on the site over the last week:

  • Implemented Squid caching for a number of the most performance intensive places on the site, including work index pages. For the biggest impact, we focused on caching the pages which are delivered to logged-out users. This is because all logged-out users usually see the same things, whereas logged in users might have set preferences (e.g. to hide warnings) which can't be respected by the cache. We initially implemented Squid caching for individual works, but this caused quite a few bugs, so we've suspended that for now while we figure out ways of making it work right. (You can read more about what Squid is and what it does in Release Notes 0.8.17.
  • Redistributed and recalibrated our unicorns (which deliver requests to the server and retrieve the data) to make sure they're focused on the areas where we need them most. This included setting priorities on posting actions (so that you're less likely to lose data when posting or commenting), increasing the numbers of unicorns, and adjusting the time they wait for an answer.
  • Simplified bookmark listings, which were using lots of processing power. We'll be looking into revamping these in the future, but right now we've stripped them back to the basics to try to reduce the load on the site.
  • Cached the listing of guest kudos so the number doesn't have to be fetched from the database every time there are new kudos (which caused a big strain on the servers)

Implementing these changes has involved sustained work on the part of our sys-admins, coders and testers; in particular, the Squid caching involved a great deal of hard work in order to set up and test. Several members of the team worked through the night in the days leading up to the weekend (when we knew we would have lots of visitors) in order to implement the performance fixes. So, we're disappointed that the changes so far haven't done as much as we'd hoped to get rid of the performance problems - we were hoping to be able to restore site functionality quickly for our users, but that hasn't been possible.

What we're going to do next

Although the emergency fixes we've implemented haven't had as much impact as we'd hoped, we're confident that there are lots of things we can do to address the performance problems. We're now working on the following:

  • New search and browse code. As we announced in our previous post on performance issues, we've been working for some time on refactoring our search and browse code, which is used on some of the most popular pages and needs to be more efficient. This is almost ready to go -- in fact, we delayed putting it onto our test archive in order to test and implement some of the emergency fixes -- so as soon as we have been able to test it and verify that it's working as it should, then we will deploy this code.
  • More Squid caching. We weren't able to cache as many things as we'd initially hoped because the Squid caching threw up some really tricky bugs. We're continuing to work on that and we'll implement more caching across the site once we've tested it more thoroughly.
  • More servers. We're currently looking at purchasing a more robust database server and moving our old database server (aka 'the Beast') into an application slot, giving us three app servers. We'll also be upgrading the database software we use so that we can make the most of this server power.

When we'll be able to implement the fixes

We're working as fast as we can to address the problems -- we poured all our resources into the emergency fixes this week to try to get things up and running again quickly. Now that we've implemented those emergency fixes, we think that we need to focus on making some really substantive changes. This means we will have to slow down a little bit in order to make the bigger changes and test them thoroughly (to minimise the chances of introducing new bugs while we fix the existing problems). Buying servers will also take us some time because we need to identify the right machines, order them and install them. For this reason, we expect it to take at least two weeks for us to implement the next round of major fixes.

We're sorry that we're not able to promise that we'll fix these problems right away. We're working as hard as we can, but we think it's better to take the time to fix the problems properly rather than experimenting with lots of emergency fixes that may not help. Since the AO3 is run entirely by volunteers, we also need to make sure we don't burn out our staff, who have been working many hours while also managing their day jobs. So, for the long term health of the site as a whole, we need to ensure we're spending time and resources on really effective fixes.

Invitations and the queue

As a result of the increasing demand for the site, we're experiencing a massive increase in requests for invitations: our invitations queue now stands at over 17,000. We know that people are very disappointed at having to wait a long time for an invitation, and we'd love to be able to issue them faster. However, the main reason we have an invitations system for creating accounts is to help manage the growth of the site -- if the 16,000 people currently waiting for an invitation all signed up and started posting works on the same day the site would definitely collapse. So, we're not able to speed up issuing invitations at this time: right now we're continuing to issue 100 invitations to the queue each day, but we'll be monitoring this closely and we may consider temporarily suspending issuing invitations if we need to.

Until recently, we were releasing some invitations to existing users who requested them. However, we've taken the decision to suspend issuing invitations this way for the present, to enable us to better monitor site usage. We know that this will be a disappointment to many users who want to be able to invite friends to the site, but we feel that the fairest and most manageable way to manage account creation at present is via the queue alone.

What can users do?

We've been really moved by the amount of support our users have given us while we've been working on these issues. We know that it's incredibly annoying when you arrive at the Archive full of excitement about the latest work in your fandom, only to be greeted by the 502 error. We appreciate the way our users have reached out to ask if they can help. We've had lots of questions about whether we need donations to pay for our servers. We always appreciate donations to our parent Organization for Transformative Works, but thanks to the enormous generosity fandom showed in the last OTW membership drive, we aren't in immediate need of donations for new servers. In fact, thanks to your kindness in donating during the last drive, we're in good financial shape and we're able to buy the new server we need just as soon as we've done all the necessary work.

As we've mentioned a few times over the weekend, we can always use additional volunteers who are willing to code and test. If this is you or anyone you know, stop by Github or our IRC chat room #otw-dev!

There are a few things users can do when browsing which will make the most of the performance fixes we've implemented so far. Doing the following should ease the pressure on the site and also get you to the works you want to see faster:

  • Browse while logged out, and only log in when you need to (e.g. to leave comments, subscribe to a work, etc). Most of our caching is currently working for logged-out users, as those pages are easier to cache, so this will mean you get the saved copies which come up faster.
  • Go direct to works when you can - for example, follow the feeds for your favourite fandoms to keep up with new works without browsing the AO3 directly, so you can click straight into the works you like the sound of.

Support form

Our server problems have caused some problems accessing our support form. If you have an urgent query, you can reach our Support team via the backup Support form. It's a little more difficult to manage queries coming through this route, so we'd appreciate it if you'd avoid submitting feature requests through this form, to enable us to keep on top of bug reports. Thanks!

Thank you

We'd like to say a big, big thank you to all our staff who have been working really hard to address these problems. A particular shoutout to James, Elz, Naomi and Arrow, who have been doing most of the high level work and have barely slept in the last few days! We're also incredibly grateful to all our coders and testers who have been working on fixing issues and testing them, to our Support team, who have done an amazing job of keeping up with the many support tickets, and to our Communications folk who've done their best to keep our users updated on what's going on.

We'd also like to say a massive thank you to all our users for your incredible patience and support. It means so much to us to hear people sending us kind words while we work on these issues, and we hope we can repay you by restoring the site to full health soon.

A note on comments: We've crossposted this notice to multiple OTW news sites in order to ensure that as many people see it as possible. We'll do our best to keep up with comments and questions; however, it may be difficult for us to answer quickly (and on the AO3, the performance issues may also inhibit our responses). We're also getting lots of traffic on our AO3_Status Twitter! Thanks for your patience if we don't respond immediately.

Comment

Release Notes 0.8.17

Published: 2012-06-09 01:42:50 -0400

Welcome to our third Release in this week! Elz, James, and Naomi contributed code to this release, and Ariana, bingeling, Enigel, Jenn, and Kylie from our testing teams worked it over. Our sysadmins and coders have done more work to address the performance issues that have been affecting the archive as well as several other bugfixes.

PLEASE NOTE: in the name of drastically improving performance, this deploy may have a few side effects that appear at first to be errors or confusing! Please do read over these release notes and make sure that they don't cover a problem you are experiencing before you contact support.

Further efforts to battle the 502 errors!

This release includes caching of most pages for guests using Squid! Squid will serve up saved versions of pages without hitting our database or application, which increases speed and decreases server load for everyone.

The tricky part is making this work with all of the dynamic elements of the site: skins, content that gets updated by users, personalized messages, etc. We have decided to turn squid on quickly to keep the Archive running smoothly but we'll be working on finding the right balance between customization and performance as we go forward, so you may see some tweaks to different aspects of the site as we fine-tune this.

Current issues related to the caching:

  • Site skins have been disabled for logged out users for the time being - if you rely on this feature for accessibility needs, please contact support and we will get you an account ASAP so you can use the skins again.
  • Comments and kudos from guests may not show up at once for other guests. When a guest leaves kudos or posts a comment, they will see the comment/kudos added. If another guest then visits that same page (or the same guest reloads the page), however, they will see the most-recently-cached version, which may not yet show their comment/kudos count.
  • Guests may occasionally see a stray error message or notice appearing at the top of a page that does not appear to be related to anything you've done. We are working to track all of these errors down but it is hard to be sure we've gotten them all. The messages should not affect using the archive.
  • Hits that are handled by Squid (most hits from guests) will not appear in the hit count immediately. The hit counts will be updated once a day from the squid logs.
  • Duplicate hits from the log files (for instance on page reloads by the same guest) will no longer be removed because of technical limitations, so hit counts may increase more quickly in some cases.

Squid will be enabled after we update the code, so you may not notice any changes right away.

For those interested in knowing more about Squid, see the detailed explanation below!

Changes to Subscription emails

We've gotten feedback about how people use their subscription emails and in response we have adjust the subject lines and message content to allow people to identify the content more easily. Emails will now contain subs of one type (author, series, or work) and the name/title of the first one in the subject together with the number of other updates.

Details

  • Subscriptions:
    • Email subjects will now say [AO3] instead of [Archive Of Our Own].
    • Subscriptions will be bundled by type with subject lines of the form [authors] posted [first item] and [#] more, where first item will be one of: [Work Title], [Chapter Title] of [Work Title], [Work Title] in [Series Name].
  • Performance:
    • Skin chooser is turned off for logged out users.
    • Nearly all pages will be cached for logged out users.
    • Comment forms and other forms that are getting data for logged out users will have their details remembered in cookies and filled in by Javascript rather than remembered in the page.
  • Bug Fixes:
    • 500 errors were appearing on some work listings because of an interaction between caching and time zone conversion - this should be fixed now.

Details About Squid

Senior coder Ana has written up some helpful information about Squid for those who are curious:

"Squid is a really powerful tool that does a lot of things, but we’re using it primarily as one thing: a reverse-proxy cache. A reverse-proxy cache is a system designed to cache (that is, store copies of) web pages. It sits between users’ requests and the rest of the site and stores the responses to some requests so that instead of making the server build the page from scratch again, Squid can check to see if someone’s looked at that page recently and pass on the cached version. This is really useful when you want to send the same page to lots and lots of users because it means that instead of forcing the servers to generate the pages over and over, we can store a copy and give that copy out to everyone.

Of course, sometimes pages change: an author edits a story, or someone leaves kudos, so you don’t want to let Squid keep those copies around forever. Right now we let Squid keep copies for 20 minutes, and then it throws them away and gets a new one. This feels like the right balance between keeping things up to date, but not overloading the servers.

In addition, logged in users get customization on every page, in the form of the user bar at the top of the page if nothing else, which means that we don’t want Squid to store or give pages to logged in users. If it did, then every user would see the user bar for whoever made the request that Squid saved, and it would only change every twenty minutes.

This same principle holds true for all on-page customization (such as the skin-chooser), and finding the right balance between customization and cacheability (how suitable a page is for storing and giving out to everyone) is going to be an ongoing project as we try to weigh site performance against nifty features and information."

These release notes written and compiled by Ana, Claudia, Elz, Enigel, Jenn, Lucy, and Naomi.

Comment

Release Notes 0.8.15-0.8.16

Published: 2012-06-07 15:21:07 -0400

Mini-release notes: Battling the 502s errors and responding to user feedback

Our coders and sys-admins have been working hard to deal with the performance issues we've been experiencing over the last few weeks. Releases 0.8.15 and 0.8.16, only a few days after deploying 0.8.14, see the introduction of several tweaks to our code and server setup that should help alleviate the site slowness we addressed in our recent post, AO3 performance issues.

New code by andreja, Ariana, Elz, Enigel, and Naomi. Tested by Elz, Enigel, Jenn, Kylie, Sarken.

Performance fixes

As noted on our Known Issues page, one of the major bottlenecks were tag listings for very popular fandoms, which would either load very slowly or throw up 502 errors. We've added caching for the first five pages of results - these will expire when a work using that tag is posted or revised, so the listings will still be up to date.

Another significant bottleneck were the bookmarks listings, both the main one and the bookmarks on tags. We've simplified the functionality, as we look into ways of reworking the bookmarks for a much better performance.

With the help of New Relic, a web service that monitors and analyzes site performance in great detail, we found another source of slowness: works with a large amount of (guest) kudos. Showing an updated kudos count for every work as it was accessed was putting an undue strain on the server, so for now the number of kudos on a work will be cached (i.e. not fetched from the database in real time) and only updated every five minutes.

As per a user suggestion (thank you, fydyan), we also looked into ways to prioritize certain user actions over others, so that trying to post a work or a comment would be less likely to throw up an error (potentially taking all input with it) than, say, accessing a user profile or browsing the site. Many thanks to Sidra from Systems for implementing this!

Invitation emails

We had a bug which was preventing notification emails for invitations being sent. We've fixed this bug, and the emails which were affected have been resent. This may result in some people receiving their invitations twice - invitations can only be used once, so please take note of the invitation code and don't pass the extra email along to a friend if it's a code you've already used!

Plans for our next few deploys

We're continuing to work hard on performance fixes and will soon be implementing much more caching across the site. We'll be posting with more details about this shortly.

We've had lots of feedback about our recent changes to notification emails. Unfortunately, we cannot roll back these changes for performance reasons (for more details see our post on Email changes and USER STATS!). However, based on feedback we've received so far, we will be adjusted the way subscription emails are batched and labeled, hopefully in the next deploy. Thanks for bearing with us while we work to improve this!

Details

  • Bug fixes:
    • Invite emails were not being sent; this has now been fixed and the delayed ones resent.
    • The work count on the front page used to vary depending on whether the user was logged in; it now shows the total of all published works.
    • Previewing a work before posting made the wordcount not show; this has been fixed.
  • Improvements:
    • Kudos counts on works are now cached, but will change when new kudos is added by logged in users. Guest kudos counts will update every five minutes. Thus it will cause less of a load on the database.
    • Works listings under tags are now cached up to the fifth page. The listings will update when a work with that tag is posted or revised.
    • We've removed the grouping of bookmarks by work. Also, before it was trying to get the bookmarks using tag synonyms, or bookmarks that were not using the tag directly. The main bookmarks listing now grabs the most recent public bookmarks on the site, and the tag listings show the bookmarks that are tagged directly with the requested tag.
    • Text on the collections page has been changed to clarify what characters are allowed in collection names.
    • Small change to text on the tag edit pages: the label for the synonym field was changed to "Choose an existing tag or add a new tag name here to create a new canonical and make this tag its synonym." for the sake of clarity.
    These release notes written and compiled by mumble, Enigel, Jenn and Lucy.

Comment

Survey Sunday #2 - membership and donations!

Published: 2012-06-03 16:53:16 -0400

This week on Survey Sunday, we're taking a look at some illuminating stats regarding OTW membership: why fans join the OTW, why fans don't join, and why fans don't renew their membership. As always throughout this series, we are only offering aggregate results and summary comments rather than individual responses.

Today we're going to focus on the following submitted question:

  • "I'm interested in the answers to 73 and 74, and to a lesser extent 72. Basically I want to know what the roadblocks are that are preventing people from donating or that are making people decide to quit donating."

Before we show you those results, however, let's talk about data we're not showing you yet. In the last Survey Sunday, we fielded several queries from some of you regarding the Fanlore sections of the survey, including a great request for conclusions drawn from cross-referencing responses to multiple questions. We're currently working on those questions and eventually hope to provide you with all of that info and more, but there were too many essay-style responses for us to complete the Fanlore section for this Survey Sunday.

How we're working on the data

The survey workgroup in charge of combing through all ~6,000 responses has been organising and filtering the data almost from day one (that is, since the results came in), but only recently met for the first time to define goals and settle on a methodology. We agreed that before we can really know how to categorize the data we're dealing with, we need to get a clear overview of what the feedback actually is.

Since many questions can be grouped together by subject, we've split up into mini-teams that have agreed to work on each grouping, both individually and as an overall composite set of data, such as the 3 questions we're sharing with you today. Here's a bird's-eye view of the organization process for these mini-teams:

  1. preview the data and develop a good understanding of it
  2. report back to the group, share results with the relevant committees, gather feedback
  3. develop a group-wide organizational system
  4. work within the team to sort and categorize aggregate and open-ended results for each question

Once these steps are complete for all questions, we plan on digging into the guts of Surveymonkey's Export options, and possibly unleashing an Excel/SQL subgroup to work database magic and expand that preliminary analysis. At this level, we'll have the ability to cross-reference questions, filter by respondent data (like country or platform), and more. But we're not there yet! As we stated previously, we hope that sharing some preliminary data publicly as we proceed will help keep you updated and actively invested in the results.

Responses to questions on OTW membership

So, let's start with a look at 3 questions regarding OTW membership to give you an overview of what the OTW looks like in terms of donor motivation.

The lead-up question to this section was #70, "Have you ever been an OTW member?", serving as the gateway determining which follow-up questions were asked. Of the 5021 respondents, 730 were currently OTW members and proceeded to answer question 72, "What was your main reason for becoming a member?", but skipped questions 73 and 74. 226 had let their membership lapse and got question 73, "What made you stop being a member?", followed up by #74, "What might make you choose to be a member in the future?". This was also the question that the 4065 respondents were routed to who answered that they had never been members.

screenshot of a graph for the question Have you ever been an OTW member. 5021 answered, 965 skipped this question. Options: yes currently, 730 replies = 14.5%. yes but lapsed, 226 replies = 4.5%. no never, 4065 replies = 81%

(Please note that the replies to question 70 don't actually reflect current membership numbers: 965 people skipped the section entirely, and the number of members had increased during the April membership drive.)

Question #72: "What was your main reason for becoming a member?"

screenshot of a graph for the question What was your main reason for becoming and OTW member. 715 answered, 5271 skipped this question. Options: I wanted the right to vote in the elections, 52 replies = 7,3%. I wanted to support the organization financially, 343 = 48,0%. I wanted to show my general support for the organization, 314 replies = 43,9%. I joined because my friends are also members, 5 replies = 0,7%. I liked the incentive merchandise, 1 reply = 0,1%. Other (please specify), 50 replies

 

As you can see, the main motivation for respondents in choosing to donate to the OTW is support. "Support," of course, is a broad concept. In the open response portion of the question, a number of recurring themes emerged:

  • respondents wanted to do a combination of the options given to them, e.g. show general and financial support, or some form of "all of the above"
  • respondents wanted to support the OTW's legal advocacy, the Archive of Our Own, and/or other OTW projects
  • respondents wanted to have a vote in the OTW elections and a voice in the direction of the organization
  • respondents just wanted to give money to the OTW, but didn't realize they would become members as well/didn't realize they had a choice about also becoming a member
  • respondents donated as a way of making up for not having time or ability to volunteer
  • respondents donated as a way to express their belief in the value of the OTW

It bears repeating that all of this is a preliminary, broad overview of trends we saw in responses; as we categorize data we will look more closely at specific motivations people have for starting, renewing, and discontinuing their OTW membership.

The same principle goes for membership renewal, which was explored further in question 73.

Question #73: "What made you stop being a member?"

screenshot of a graph for the question What made you stop being a member?. 211 answered, 5775 skipped this question. Options: I forgot to renew membership, 90 replies = 42,7%.	
Financial reasons, 86 replies =	40,8%. I lost interest, 6 replies = 2,8%.
I feel there is a lack of progress, 3 replies = 1,4%. I don't feel represented by the OTW, 9 replies = 4,3%. I disagree with (aspects of) the OTW's policies or their development, 17 replies = 8,1%. Other (please specify), 22 replies.

 

It's important to remember that when we designed the survey, we included these particular options because we realize that acknowledging former member dissatisfaction is crucial to learning how to increase membership renewal. It's heartening to know that 42% of respondents might potentially renew their membership, but we also care about the 13-17% of former members who had negative or ambivalent experiences with the OTW after they donated to it. We appreciate these respondents, both for their initial memberships, and for taking the time to share their experiences as members with us.

It's harder to pick up on general themes for this question, especially due to the more contentious nature of the question, and the fact that we had fewer open responses to this question. However, some recurring trends from respondents were:

  • A combination of two or more of the closed-ended responses
  • Former members plan to renew but just haven't gotten around to it
  • Former members shifted their support to volunteer work instead for numerous reasons
  • Former members had negative experiences or received negative impressions during OTW election campaigns, or during their time as members

We hope to cross-analyze these types of responses with other responses to different aspects of the survey to glean more specific data about the things that hinder and help our efforts in this area.

Question #74: "What might make non-members join in the future?"

screenshot of a graph for the question What might make you choose to be a member in the future?. 3324 answered, 2662 skipped this question. Options: Nothing, the OTW is just not for me, 556 replies = 16,7%. Improvement in my finances, 1972 replies = 59,3%. Niftier incentives, 112 replies = 3,4%. Better progress on existing projects, 93 replies = 2,8%. Outreach to my fannish community, 205 replies = 6,2%. Services or tools that appeal to me or my fannish community, 336 replies = 10,1%.	Change in the organization's policies, 50 replies = 1,5%. Other (please specify), 484 replies.

If long-term and new membership are crucial to our health as an organization, then it doesn't take a team of SQL experts to see right away that there's a statistical correlation between the 13-17% of lapsed members from Question #73 who are unsatisfied with our organization's current fannish outreach, general policies, direction, and progress, and the 17-21% of potential members who cite solutions to these same issues as potential incentives to join.

However, this set of questions weren't designed to ask what creates levels of dissatisfaction—only to quantify the relationship between dissatisfaction and membership. Other parts of the survey will hopefully provide more context. What this data does provide us is a sharp, bird's-eye perspective on the rest of the survey: every area where we succeed or fall short, no matter how small, connects back to our actual membership numbers.

And while, alas, we can't control the state of your finances and we definitely understand that the OTW is not for everyone, there are many things we can change to impact membership for the better. We hope the survey analysis will ensure that the organization as a whole knows what we need to change and renews its commitment to do so at every level.

What we can't tell you right away, though, is what those numbers mean in terms of conclusions or future direction. Once our Development & Membership committee has had time to familiarize themselves with these results and discuss them in depth, we will be able to amend this report with their interpretations and conclusions. We will collect similar analyses from committees along with creating more reports, and compile it all into a more fleshed-out final report.

In the meanwhile, check our list of all survey questions and tell us what you're curious about for the next Survey Sunday post on June 17th!

Mirrored from an original post on the OTW blog. Find related news by viewing our tag cloud.

Comment

Release notes for release 0.8.14

Published: 2012-06-03 12:43:54 -0400

Welcome to Release 0.8.14. Ariana, Elz, Enigel, Lal, mumble, Naomi, Sam J. and Sarken contributed code to this release, which was tested by our awesome testing team: Jenn, Kylie, Melanie, mumble, Sparkling Snowfall, Xparrot and Zebra. A special welcome to our new coders Andreja, Ariana and Lal, who already contributed their first bug fixes for this deploy and the next! \o/

Right on the heels of our April deploy come the changes to our notification emails and the stats feature that we had previously announced. We welcome feedback on both! You can comment here, on the announcement or, for those more comfortable with code branches and spec drafts, our Github wiki.

We also spent some time going through our bug tracker with a fine comb and flagging several older issues that had been lingering for a few months or years. Some of those had been magically fixed somewhere along the way, some weren't relevant to our current code base anymore, some simply needed a confirmation that the fix worked as well on the actual Archive as it did in our test environment. Consequently, 53 additional issues could be closed in this deploy cycle.

In our next deploy, we're hoping to introduce some major improvements to our search and browse system. These improvements should help us address some of the performance problems we've been experiencing recently. We'll be testing everything carefully to ensure that we don't make the Archive blow up, but if it all works out, that will be coming your way within the next month. We'll have a preview for you before it's released, so stay tuned for more news!

Highlights!

Stats page!

This deploy sees the launch of the first version of our user stats page! Senior coder Naomi incorporated several popular feature requests into this information-rich overview, and added a few more features based on the feedback you gave us when we posted a preview of the stats page. You can see your total number of subscribers, kudos, and hits as well as your total word count, and a list of your works sorted by hits, kudos, bookmarks, or comments. On top of that, all this information can also be broken down by year!

Please note that this feature is still in progress, and improvements and bug fixes are in the works. Due to the way subscriptions are currently handled, it's difficult to differentiate between subscriptions to a work and to the series this work is part of. Consequently, series subscriptions are not being counted right now. Please stay tuned for updates!

A note about referrer information: The next version of the stats page will include a short list of recent referrers for each work. If you're unsettled by the idea that an author can see that you linked to their work, for example in a Tumblr or Dreamwidth post, you can use a go-between like anonym.to to link to works on the Archive. The content of locked or filtered posts will obviously not be accessible even if the link to it is listed! Furthermore, the mere fact that you linked to a work in the past will not make your post show up in the referrers. Someone has to click on that link as well.

Batch emails!

We know that many of our users liked to receive kudos notifications in real time, as a little nibble of joy throughout the day or week. However, the sheer number of emails going out every minute was bringing our server to its knees, so we had to drastically cut down on our email traffic. :-(

As a result, kudos notifications will go out only once a day, and subscription emails (new works by an author, new chapters in a WIP, or new parts in a series) will be batched and sent out hourly. This will relieve the pressure on our servers and help ensure emails arrive in a timely fashion. We're open to feedback on this process, and are hoping that the stats page will at least partly make up for the change!

Tools for Open Doors staff!

Open Doors gets an arsenal of new tools to make the importing process smoother, such as the ability to resend notification emails for unclaimed authors, for users that accidentally deleted it the first time, or no longer have access to the email address registered with the imported archive. (Previously, manually transferring works to current AO3 users generated a co-author notice which in turn generated confusion; for new users, invitations to the Archive needed to be sent separately.)

The Open Doors committee will also be able to change the redirect URL for works on the imported archive's original site, which is handy for users who have accidentally deleted the imported copy of their works or want to redirect readers to an existing copy of the work on AO3. (This is an interim fix until the merge tool is developed, which will allow authors to pick one version of the work to keep, and combine hit counts and reviews of both works.)

Note: If this is something you want Open Doors to arrange for your Smallville Slash Archive imports, please use this contact form to get in touch. Make sure to include your AO3 username as well as links to your work(s), both here and on the original archive. Open Doors staff will take care of it!

Last but not least, Open Doors will also be able to quickly look up what works have been imported under a particular email address, and block works associated with a particular email address from ever being imported, at an author's request.

Known Issues

We're currently experiencing some significant performance problems. We're working on the issue - now that we have deployed the code from this release we're able to start testing the big changes to search and browse, which should help a lot. You can read more about the performance problems and what we're doing to fix them at our post on AO3 performance woes

See our Known Issues page for current issues.

Note: This page recently underwent a thorough update, as well as a restructuring of the available information for better scanning. We have added workarounds where possible and will keep adding information as it becomes available.

Release Details

Features

  • Stats page
  • Email batching
  • Mass Import improvements

Bug fixes

  • Importing
    • The code to manage redirects from imported collections to works on the Archive was optimized to better handle valid and invalid URLs
    • Tweaked the wording of emails sent out to authors whose works were imported by an archive mod, it's now clearer that initially restricted (members-only) works can be unlocked by the author or archivist
    • Trying to import a work with one or more chapters longer than our limit (500,000 characters in a chapter) caused imports to just fail - as a stopgap, the imported work is truncated and a warning message is added at the end
    • Some invitations to claim imported stories were not being delivered properly; these should now go out without a hitch
    • Stories by two co-authors should now import correctly: both authors should receive an AO3 invitation and be able to claim the story as theirs independently from each other
  • Works & Chapters & Comments
    • On multi-chaptered works, the word count would include unposted draft chapters; this has been fixed to only count posted chapters
    • When previewing a new chapter, then going back to edit and post the chapter, the chapter would get stuck as a draft; this has been fixed
    • We've received some user reports about whole works being accidentally deleted instead of single chapters; we have now renamed the "Delete" button on work pages (which also appears when viewing a single chapter) to "Delete Work" to avoid confusion
    • The character counter for comments wasn't working when editing a pre-existing comment; this has been fixed for all form fields, including the text box for challenge instructions (where the counter wasn't working to begin with)
    • The links in comment notifications (to reply, to see the whole thread, etc.) were broken in various ways, this should be fixed now
  • Skins & Cosmetics
    • On user pages, the link to the profile was in an easily overlooked position below the user name, it has now been moved to the sidebar navigation
    • One Point Faux skin: on user dashboards, the list of top five fandoms wouldn't show up in some browsers; this has been fixed
    • One Point Faux skin: on user dashboards, the text on the Bookmark button was greyed out to discourage clicking until implementation of this feature
    • One Point Faux skin: in the header, a | symbol was appended to the "Hi, example!" greeting, this has been taken care of
    • Reversi skin: on the Manage Chapters page, the chapter titles were white-on-white and thus unreadable; this has been fixed
    • When adding several works as an inspiration (e.g. for a remix or fanart), the list of "current parent works" was misaligned; this has been fixed (the same fix applies to a similar issue with works in more than one series)
    • On prompt meme dashboards, the "Manage Items" option would show up twice for maintainers; the unnecessary duplicate has been removed now
    • Updated the help text for searches by Date on the Advanced Search form to go with a recent bug fix
    • Renamed the "Full text:" label to "Any field:" (everything else on the form plus summaries and notes), as full work searches were disabled for performance reasons a while ago
    • Changed the wording of the "only available to registered users of the Archive" message (displayed to guests without accounts) to encompass all restricted works, not just stories
    • On the Preferences page, the text on the help pop-up explaining the "Hide the share buttons on my work." option was adjusted to match the wording of the preference
  • Error Extravaganza
    • When trying to download a .pdf version of a work that hadn't been generated (yet), users would get an error 500 instead of seeing a helpful error message; this has been fixed
    • There was an issue with an old reference to Javascript code that would cause errors 500 when trying to access the tags index; this has been resolved
    • Subscriptions as well associations between a work and its author weren't properly removed from the database when a work was deleted - resulting, for example, in errors 500 when trying to manage subscriptions; this has been fixed
    • Trying to access the bookmarks index for a nonexistent (e.g. misspelled) user or tag, or dashboard pages for a nonexistent pseud or Collection would throw up an error 500; it now gives the proper error message
  • Misc.
    • The login link in the header was unreachable in keyboard tab navigation, making it inaccessible for users of screen reader software; we apologize for this and will take even greater care about keyboard accessibility in future releases
    • The directory for work downloads (such as .epub and .mobi files) was organized by author only, resulting in a gigantic list of folders and potentially slowing down performance in the future; subdirectories have been added now
    • On the Abuse Report form, the "link to the page you are reporting" field didn't recognize https:// URLs as proper Archive links; the validator now allows those as well
    • Tidied up a small bit of code for gift exchanges and prompt memes
    • Added a note about recommended measurements (100x100px) to the icon upload form for user profiles

Comment

AO3 performance issues

Published: 2012-06-01 02:38:41 -0400

As pretty much all of our users have no doubt noticed, we've been experiencing some problems with Archive loads: slowdowns and the appearance of the dreaded 502 page have become a regular occurrence. We're working on addressing these issues, but it's taking longer than we'd like, so we wanted to update you on what's going on.

Why the slowdowns?

Mostly because there's so much demand! The number of people reading and posting now is overwhelming - we're glad so many people want to be here, but sorry that the rapid expansion of the site is making it less functional than it should be.

We now get over a million and a half pageviews on an average day, often clustered at peak times in the evening (particularly when folks in the Western Hemisphere are home from work and school) - we were using a self-hosted analytics system to monitor site traffic, and we had to disable it because it was too overloaded to keep up. The traffic places high demands on our servers, and you see the 502 errors when the systems are getting more requests than they can handle. Ultimately we'll need to buy more servers to cope with rising demand, but there's ongoing work that we've done and need to continue to do to make our code more efficient. We've been working on long-term plans to improve our work and bookmark searching and browsing, since those are the pages that get the most traffic; right now, they present some challenges because they were designed and built when the site was much smaller. We've learned a lot about scaling over the years, but rewriting different areas of the code takes some time!

What are you doing to fix it?

Our Systems team are making some adjustments to our server setup and databases. Their first action was to increase the amount of tmp space for our MySQL database on the server - this has alleviated some of the worst problems, but doesn't really get at the underlying issues. They're continuing to investigate to see if there are additional adjustments we can make to the servers to help with the problems.

We're also actively working on the searching and browsing code: that's been a big project, and it will hopefully make a significant impact. Because it affects a lot of crucial areas of the site, we want to make sure we get everything right and do as much testing as we can to ensure that performance is where it needs to be before we release it. We're switching from the Sphinx search engine to elasticsearch, which can index new records more rapidly, allowing us to use that for filtering. That will offer us more flexibility, get rid of some of our slower SQL queries, and take some pressure off our main database, and it also has some nice sharding/scaling capabilities built in.

We also try to cache as much data as we can, and that's something we're always looking to improve on. Systems and AD&T have discussed different options there, and we'll be continuing to work on small improvements and see what larger ones we may be able to incorporate.

When will it be fixed?

It's going to take us a few weeks to get through all the changes that we need to make. Our next code deploy will probably be within the next week - that will include email bundling of subscription and kudos notifications, so that we can scale our sending of emails better as well. After that, we'll be able to dedicate our resources to testing the search and browsing changes, and we're hoping to have that out to everyone by the end of June. We rely on volunteer time for coding and testing, so we need to schedule our work for evenings and weekends for the most part, but we're highly motivated to resolve the current problems, and we'll do our best to get the changes out to you as soon as we can.

Improving the Archive is an ongoing task, and after we’ve made the changes to search and browse we’ll be continuing to work on other areas of the site to enable better scalability. We’re currently investigating the best options for developing the site going forward, including the possibility of paying for some training and/or expert advice to cover areas our existing volunteers don’t have much experience with. (If you have experience in these areas and time to work closely with our teams, we’d also welcome more volunteers!)

Thanks for your patience!

We know it's really annoying and frustrating when the site isn't working properly. We are working hard to fix it! We really appreciate the support of all our users. ♥

Comment


Pages Navigation