AO3 News

Site Stats: A Look at 2013 and Beyond, Part 1

Published: 2014-01-27 18:38:38 -0500

2013 was another record year for the Archive of Our Own in terms of site growth and traffic increases, and we'd like to highlight some figures in this post (and the following). All raw data used for these charts is available in spreadsheet form, if you want to crunch the numbers or create some graphs yourself.

» AO3 Stats - 2013 (on Google Drive)


More works! More kudos! More everything!

We started the year 2009 with 182 users and a little over 2,000 works, a few of which probably said something like, "dskjdlj test test," or variations thereof. The Archive code was still being written by a small group of volunteers, who created a framework for posting, browsing, and commenting on fannish works entirely from scratch. It was considered stable enough for the general public in November 2009, when we kicked off our Open Beta phase. By January 2010, we had over 4,200 users and 36,500 works.

In 2013, we went from about 103,000 accounts to 247,000, and we passed the quarter-million mark earlier this month. (Excitement!) Roughly 416,000 new works were posted to the Archive last year, which is almost half of all our current works. (And for those reaching for the calculator right now, that's about 1,140 new works per day, on average.)

The really staggering increases, however, happened where users were interacting with each other: leaving kudos, talking in the comments, adding works to their bookmarks and recs.


Month-by-month growth of kudos, bookmarks, comments, works, and users for the year 2013. The high numbers and steeper increases for kudos, bookmarks, and comments dominate the chart, with the lines for works and users seemingly crawling along the bottom. All numbers can be found in the 'works & users' tab in the linked Google Drive spreadsheet.


We went from roughly 2 million comments in our database to almost 5.5 million, and from 10 million kudos to 28 million. (That number is one of the reasons we can't offer a feature that would give you a list of all works you've left kudos on: there's just too much data to make those calculations for each individual user while keeping the servers happy.)

What really took off in 2013 were bookmarks: 1.8 million at the beginning of 2013, almost 7 million by the end.

As it turned out, our search index code was not prepared for this amount of data at this level of user activity, and slowly broke down towards the end of last year. We are currently rewriting the relevant part of our code, making use of new tools at our disposal, for a more streamlined indexing process.

To better illustrate trends in all these numbers, here's a chart showing relative bookmark, kudos, comment, user, and work growth in 2013. (Increases are shown in percent: for example, going from 50 works to 100 would be a 100% increase, and going from 100 comments to 150 would be a 50% increase.)


Month-by-month relative growth of bookmarks, kudos, comments, works, and users for the year 2013 (in percent, starting on January 1, 2013). Bookmarks show the steepest increase (277%), users the lowest (77%). Kudos and comments show an almost identical growth, even though they differ in absolute numbers. All numbers can be found in the 'works & users' tab in the linked Google Drive spreadsheet.


Over the last twelve months, kudos and comment numbers almost doubled (both increasing by roughly 190 percent), while the number of bookmarks almost tripled (280% percent increase).

One last stat we looked at were average feedback numbers per work, given the total number of works.


Month-by-month growth in the average number of kudos/comments/bookmarks per work. The numbers were calculated for every month, based on data in the 'works & users' tab in the linked Google Drive spreadsheet.


As you can see, these numbers also went up over the year. We started out with 3.6 comments, 3.4 bookmarks, and 18 kudos per work, on average. This was followed by a relatively slow increase in works, and a much steeper increase in feedback numbers. By the end of 2013, we registered 5.8 comments, 7.3 bookmarks, and 30 kudos per work. (Of course, all these are averages across almost a million works, so a relative handful of very widely-read works in popular fandoms will drive up the numbers by quite a bit.)


More traffic!

One way to measure site activity is by looking at works and comments, another is through reports from our server monitoring tool, New Relic. Among many other things, it keeps track of how many pages were generated and served to our users at any given time.

We started the last year with 27.6 million page views in the first week of January 2013, which comes out to roughly 3.9 million page views a day, or 2,700 page views a minute. That's on average, of course. Sundays will be much busier than any weekday, and there are peak times and slower periods throughout the day.

In the first week of January 2014, we counted 49.5 million page views. At this rate, a Sunday with 7 million page views isn't a rarity anymore. That's an average of 4,800 pages served to users every minute.

The following graph shows this increase in page views, focusing on 2013 specifically: The first week of January represents our zero point, and the growth is charted from there.


Increase in weekly page views during 2013, starting at the first week of January 2013, and ending at the first week of January 2014. Every Monday-Sunday period is represented by a dot, and the dots gently meander upwards, with a steeper increase towards the end of the year. Numbers are available in the linked Google Drive spreadsheet.


The pattern we witnessed in 2012/2013 - increased traffic in December, with a considerable peak in the first week of January, followed by a brief lull - also held in 2013/2014.

Even though we're only hosting text-based works right now, this much site usage still generates a fair amount of traffic to and from the database. By the end of 2013, we were moving 10 Terabytes of data every month.

As mentioned before, all these numbers (and more!) are available in spreadsheet form:

» AO3 Stats - 2013 (on Google Drive)


This concludes our first post! The next one will follow at the beginning of February and include looks at international site usage (spoiler: we are everywhere!) and browser preferences (spoiler: a lot of people browse from phones and tablets). If you have any questions or have created any additional charts, let us know!


What's up with the recent downtimes?

Published: 2014-01-23 16:26:51 -0500

If you're a regular Archive visitor or if you follow our AO3_Status Twitter account, you may have noticed that we've experienced a number of short downtime incidents over the last few weeks. Here's a brief explanation of what's happening and what we're doing to fix things.

The issue

Every now and then, the volume of traffic we get and the amount of data we're hosting starts to hit the ceiling of what our existing infrastructure can support. We try to plan ahead and start making improvements in advance, but sometimes things simply catch up to us a little too quickly, which is what's happening now.

The good news is that we do have fixes in the works: we've ordered some new servers, and we hope to have them up and running soon. We're making plans to upgrade our database system to a cluster setup that will handle failures better and support more traffic; however, this will take a little longer. And we're working on a number of significant code fixes to improve bottlenecks and reduce server load - we hope to have the first of those out within the next two weeks.

One area that's affected are the number of hits, kudos, comments, and bookmarks on works, so you may see delays in those updating, which will also result in slightly inaccurate search and sort results. Issues with the "Date Updated" sorting on bookmark pages will persist until a larger code rewrite has been deployed.

Behind the scenes

We apologize to everyone who's been affected by these sudden outages, and we'll do our best to minimize the disruption as we work on making things better! We do have an all-volunteer staff, so while we try to respond to server problems quickly, sometimes they happen when we're all either at work or asleep, so we can't always fix things as soon as we'd like to.

While we appreciate how patient and supportive most Archive users are, please keep in mind that tweets and support requests go to real people who may find threats of violence or repeated expletives aimed at them upsetting. Definitely let us know about problems, but try to keep it to language you wouldn't mind seeing in your own inbox, and please understand if we can't predict immediately how long a sudden downtime might take.

The future

Ultimately, we need to keep growing and making things work better because more and more people are using AO3 each year, and that's something to be excited about. December and January tend to bring a lot of activity to the site - holiday gift exchanges are posted or revealed, people are on vacation, and a number of fandoms have new source material.

We're looking forward to seeing all the new fanworks that people create this year, and we'll do our best to keep up with you! And if you're able to donate or volunteer your time, that's a huge help, and we're always thrilled to hear from you.


Help the OTW Grow! Volunteer Today

Published: 2014-01-20 12:16:18 -0500

Banner by Erin of a close-up of Rosie the Riveter's arm with an OTW logo on it and the words 'OTW Recruitment'

Today, we're excited to announce the opening of applications for:
  • Development & Membership Committee Staffer - 27 January 2014
  • Translation Committee Staffer - 27 January 2014
  • Communications Committee: AD&T Liaison - open until filled

We have included more information on each role below. Open roles and applications will always be available at the volunteering page. If you don't see a role that fits with your skills and interests now, keep an eye on the listings. We plan to put up new applications every few weeks, and we will also publicize new roles as they become available.

All applications generate a confirmation page and an auto-reply to your e-mail address. We encourage you to read the confirmation page and to whitelist volunteers -(at)- transformativeworks -(dot)- org in your e-mail client. If you do not receive the auto-reply within 24 hours, please check your spam filters and then contact us.

If you have questions regarding volunteering for the OTW, check out our Volunteering FAQ.

Development & Membership Committee Staffer
The Development & Membership committee (DevMem) coordinates the OTW's fundraising and membership-building activities. Our work includes planning membership drives, designing a convention outreach process, answering questions from donors, and exploring new fundraising opportunities. If you have skills and interests in fundraising, writing or designing promotional content, or customer service, consider applying to join our staff.
Applications are due 27 January 2014

Translation Committee Staffer
The Translation Committee coordinates localization efforts across the OTW: the translation of site pages, news posts, Archive FAQs, AO3 Support tickets, and any inquiry that reaches a committee or volunteer group in languages they can’t translate themselves. While volunteers do the actual text translation and editing, staff helps them by keeping track of priorities, deadlines and pending tasks; assigning work; talking and working with other committees to arrange tools for the volunteers; uploading translated documents; writing training and procedures documentation; and many other tasks involved in managing a wide, diverse and very active volunteer pool.

If you enjoy teamwork, if you like organizing, writing documentation and working with people, if you’re passionate about the OTW and its projects, and want to help us reach more fans all around the world, working with Translation might be for you!

Applications are due 27 January 2014

Communications Committee: AD&T Liaison

Communications staffers are responsible for the distribution of information internally to OTW personnel and externally to the general public, the media, fans, and other fannish organizations. Communications is also typically the first point of contact for someone interested in or wanting help from OTW.

The position of AD&T liaison requires an individual with an interest in technical writing to help maintain that area of our communications and outreach strategy. If you are interested in the behind-the-scenes development of an open-source project, and would like to help us create and share news about the AO3, this is the position for you!
Applications will be accepted until the role is filled.


AO3 Mail Bag: PM Feature

Published: 2013-12-31 20:51:21 -0500

Random Letter #2: How can I contact other users? Do you have a PM feature?

This is a question we get asked a lot, and we're always sorry to say that no, you can't message someone privately on the Archive yet. The feature is listed on our Roadmap, but it'll be a while yet before we can offer PMs.

On the one hand, we want to carefully consider how the new messages would fit into the current inbox layout and which features are needed right from the start to ensure user privacy and prevent abuse. We also need to optimize our mail server settings and code to handle even more emails (we currently send out around 880,000 emails per week) without getting blocked by spam filters.

On the other hand, we want to sit down with our Abuse and Support teams to come up with rules and policies to handle, for example, reports of harassment through private messages. We will also draw on their expertise in designing the feature, as they are the ones who most interact with our users.

Until we have built-in messaging, we recommend including some kind of contact information in your profile (the link to a user's profile can be found right at the top of the Dashboard / Works / Bookmarks / Gifts etc. links at the left):

If you're logged in and looking at your own profile, you will see a "Edit My Profile" button at the bottom. Once clicked, you can add links and information to your "About Me" section: your Twitter account, your Tumblr, your Dreamwidth, or anything else you feel comfortable sharing. Keep in mind that this information will be visible to everyone, not just logged-in AO3 users.

If you have any more questions, let us know via a Support ticket!


Release Notes for Release 0.9.12

Published: 2013-12-21 09:00:28 -0500

Welcome to Release 0.9.12! Cecilia, Elz, Emily E., Enigel, James_, Karangunii, Sarken, Scott, Stephanie, and tuff_ghost, contributed code to this release, which was tested by our awesome testing team: Camilla, Enigel, JanOda, Lady Oscar, MilenaDaniels, mumble, Runt, Sarken, and Sumeria.

This is our last deploy of this year, and we sincerely thank everyone who contributed to this or previous releases, in whatever small or big ways. Every line of code, every minute of testing, every supportive cheer helps this project prosper.

A look back at growth stats for this year will be posted in January, for all fans of hard data and color-coded charts. The Accessibility, Design & Technology team will take a short break over the holidays, but will stand by for any emergencies. We have big plans for 2014 and will share any news with you as we start new adventures. See you next year!


Backend improvements and server work

We've had an interesting couple of months, with growing traffic and increased server load, eventually leading to a collapse of our search index. You can find details about this in our updated news post, Site slowness and other currently known issues. We've been doing a lot of server maintenance in the past few weeks to tweak things, correct some settings, and generally make sure that the site will be running more smoothly. A million thanks to our all-volunteer Systems team, and especially sysadmin James_, for all their hard work! ♥

One issue we've consistently run into is with the code that refreshes the search index for works and bookmarks. This happens, for example, when a tag is edited by our wrangling team to reflect a new tag relationship or a new character name. Since we have a lot of works and bookmarks, updating the index to reflect all changes was taking a lot of time. More often than not, one re-index run wasn't even done when a new one was started.

This deploy includes a code change in how we handle these re-indexing tasks. They are now run by a tool called Resque, which we already use for many other tasks on the site that involve a queue (e.g. all subscription notifications going out at a certain time). This change allows us to monitor the re-indexing process and throw more power at it if necessary.

We are hopeful that this change will alleviate some of our problems during the traffic-heavy holidays, but we will also continue to look into further improvements both in our code as well as our server settings.

Known Issues

See our Known Issues page for current issues.

Release Details

Bug fixes

  • Works & Bookmarks
    • Work and bookmark re-indexing is now being queued and run in the background using Resque, making it more robust and less of a strain on the servers.
    • Backdating wasn't working correctly when posting a work without previewing first. This has now been fixed.
    • Unposted drafts (created when previewing) had no revision date, causing errors in several places on the Archive, such as item management pages for collections. We fixed this recently, and are making the fix permanent now.
    • The "Works (#)" button on user dashboards (below the five most recent works) had accidentally vanished in a previous deploy. We have now restored it and also added "Works (#)" and "Bookmarks (#)" buttons to collection dashboards for consistency.
    • The "↑ Top" button on a work page was forcing a full page refresh; it now just jumps you back to the top of the work as intended.
    • The gifts page now displays gifts a user has received in order from most recent to oldest.
    • Searches with slashes in them (i.e. all relationship tags) were giving Error 500 after a recent Elasticsearch update. We fixed this problem shortly after being alerted to it; this deploy makes the fix permanent.
  • Downloads
    • PDF and MOBI downloads were broken; we quickly fixed them after the last deploy and are making the fix permanent in this one.
    • Trying to download a MOBI file directly from a Kindle device would frequently lead to errors; we have changed the MIME type of our MOBI files so the Kindle browser can handle it correctly. (Due to another issue, multi-chapter works may need to be downloaded twice to get a functioning file.)
  • Tag Wrangling
    • If a tag wrangler ticked the "Turn off emails about comments" option in their account preferences, they would also no longer receive comment notifications from tag discussions. They will now always receive such comments.
    • Updating an unsorted tag from a page other than the first would redirect you back to page 1; this no longer happens.
    • There was a small display issue with autocomplete fields on tag editing pages; this is now fixed.
  • Misc.
    • The account creation page advised newly signed-up users to contact Support if the activation email hadn't arrived within two hours. We adjusted the time to 24 hours to account for clogged mail queues or issues with some email providers, which are delaying email delivery.
    • Our maintenance page (sometimes used for downtime during a deploy) was promising new features after the downtime; this wasn't always true, so we removed the line.
    • On our Invite Requests page, we were only displaying the number of people already in the queue. We have added the number of invites we automatically send out every day, which currently makes for a wait time of 24 hours or less.
    • On our Public Skins page, the skin descriptions were frequently one unbroken line, forcing sideways scrolling. In user inboxes, comments containing long unbroken words were also breaking the page. We have now added a fix that forces linebreaks. (In some browsers the lines won't be wrapped in the inbox; in that case a scrollbar at the bottom of the comment list should make it possible to read anyway.)


Ways to support the OTW this December

Published: 2013-12-05 19:07:10 -0500

2013 is almost over, and all of us at the OTW are grateful for the tremendous support shown by our members and donors this year. If you've been thinking about donating to the OTW but haven't done so yet, you may want to take a look at your finances and see whether it would be to your benefit to do so before December 31. Donations to the OTW are tax deductible in the United States. And if you're employed, please find out if your employer offers matching donations! Every dollar you give could be worth two dollars to the OTW.

Cover of the book 'Fic: Why Fanfiction Is Taking Over the World'

Another way to support the OTW this month is through our ongoing promotion with Smart Pop Books, publisher of Fic: Why Fanfiction Is Taking Over the World . Edited by Anne Jamison and featuring contributions from several current and former OTW staff members, this book would make a great holiday gift for a fellow fan! From now through December 31, if you order the book from Amazon through this referral link, Smart Pop will donate a percentage of the proceeds to the OTW. They have generously offered to double Amazon's usual affiliate rate, which ranges from 6-8% depending on sale volume, so the OTW will receive 12-16% per book. This promotion applies to both the paperback and Kindle editions.

We'd like to thank everyone who's already participated in the book promotion — your purchases since October 30 have raised more than US$180 for the OTW! And thank you to all of you who have donated to the OTW throughout the year — we are very grateful for your support.

Mirrored from an original post on the OTW blog. Find related news by viewing our tag cloud.


November 2013 Newsletter, Volume 75

Published: 2013-12-03 19:56:33 -0500

Banner by caitie of a newspaper with the name and logos of the OTW and its projects on the pages.

For more information about the purview of our committees, please see the committee listing on our website.


Journal requested, and obtained, a bibliographic listing in the Modern Language Association (MLA)'s bibliography of journals. This will provide wider visibility in academia for research published in Transformative Works and Cultures. The MLA team is currently indexing 5 years' worth of TWC issues, which will take a few months.

In addition, TWC editors Karen and Kristina have edited The Fan Fiction Studies Reader, a reprint anthology that includes some crucial essays in the fan studies field. The corrected galleys have been sent to the University of Iowa press for an expected 2014 publication date!

Legal submitted formal comments to the U.S. National Telecommunications and Information Administration (NTIA) and the U.S. Patent and Trademark Office (PTO), which had requested public comments on copyright policy issues, including the legal framework for the creation of remixes. The comments incorporated many stories submitted by fans and is likely to be a rewarding read for anyone interested in fandom and fanworks.

Special thanks go to Rebecca Tushnet, Casey Fiesler and Rachael Vaughn for their work on this document. Rebecca has been asked to testify to these agencies in connection with the same public comment process and we'll report on that when it happens. Many thanks to all of the many fans who sent in their stories. We really appreciate it.

Also in November, Legal assisted the Stanford Fair Use Project with its comments in connection with the same process, and answered a variety of legal queries from fans. They've also been working with a law student volunteer to create some educational materials. Legal also provided two posts on developments in copyright law during November: one regarding a proposal to create a "small claims" procedure for copyright disputes, and another regarding a major fair use ruling in the Google Books case.

Strategic Planning is currently surveying the Development & Membership and Communications Committees. They are also writing their report on Support. Reports for the Grants Workgroup and Systems Committee are under review by Board while the Survey Workgroup report is currently being prepared for public release.

Lastly, Communications staffer John Bayard made a presentation at the Mid-Atlantic American Popular Culture Meeting and several people told him how much they enjoyed using the AO3 and the work the OTW does!


Accessibility, Design and Technology deployed Release 0.9.11 at the AO3, and are hoping to squeeze in one more code push before the end of the year. Our servers are in need of an upgrade and traffic has increased noticeably over the last few weeks (and is expected to increase even more during the holidays). As a result, we've been battling some performance and other issues and are working on fixes. Many thanks to Systems and Support for their work on these problems. Systems also brought up a new Virtual Machine server, which is hosting multiple virtual servers dedicated to supporting the Archive.

Additionally, AD&T has decided to apply for a humanities-related grant by February to pay for a good chunk of specific coding work, including an API. Many thanks to Scott for taking point on this.

Tag Wrangling assisted with testing some tag-related bug fixes in the last deploy, and the wrangling committee is currently working with AD&T to track some wrangling bugs. As well, due to the recent server issues, wrangling is currently limited to reduce strain on the servers, so the wrangling of some fandoms may be delayed until that’s resolved.

Abuse is seeing roughly 350-400 tickets a month come in. Their new recruits are settling in and should be starting on some of those tickets soon!

Open Doors held two public chats and provided information on the planned import for StargateFan, a gen Stargate SG-1 and Stargate Atlantis archive to the Archive of Our Own.

In its new home, StargateFan will be a separate, searchable collection with its own identity for all of the fan fiction, fan art, and fan comics housed on the current archive. Open Doors will begin manually importing works from StargateFan to the AO3 collection in December 2013.


Volunteers & Recruiting has been busy with final inductions for the year and reviewing the results of the OTW's annual Still Willing to Serve Survey. This allows for current volunteers and staffers to indicate if they'd like to continue on in current positions, move to new ones, or depart the OTW. Recruitment will begin again in January after VolCom has been able to finish processing departing volunteers and taken a well deserved breather!

This is also the OTW's final newsletter for 2013, but it will return in 2014 as committees begin work in the new year.

New Committee Staff: DandalfTheWhite (Abuse), Teddy Escher (Abuse), Elizabeth Young (Abuse), Pteriforever (Abuse), DragonWyrd316 (Abuse), Jess C. (Abuse), Kelsi No (Abuse), 2 other Abuse staffers, Sierra Wilson (Wiki), 1 other Wiki staffer

New Coder Volunteers: Storm

New Translator Volunteers: Agnieszka Górniok, Ania Kopertowska, Bianca, Dai-kun, Espirofito, Jocelin Potash, Katarina Harju, Lenore, Marinet252, monnwilk, Nana, nightmarefluff, Ruth Damaris, WenBunny, Whovie, 6 others

Departing Committee Staff: 1 Internationalization & Outreach staffer, Eva Bolinder (Translation), 3 other Translation staffers, Amy Luo (Communications), 1 other Communications staffer, 1 Journal staffer

Departing Tag Wrangler Volunteers: 1

Departing Translator Volunteers: Eva Bolinder, 39 others.

Mirrored from an original post on the OTW blog. Find related news by viewing our tag cloud.


Site slowness and other currently known issues

Published: 2013-11-20 14:45:01 -0500

Update December 14, 18:00 UTC: As of this week, all systems should be back to normal. We're still working on optimizing our server settings, so very brief downtimes for maintenance should be expected. If bookmarks still won't sort correctly for you - we're working on a more permanent fix to the underlying issue, but it might be a short while yet. As always, we're keeping an eye on Support tickets and messages to our Twitter account, and will react as quickly as possible if anything seems off. Thank you all for your patience.

Update December 3, 16:00 UTC: We have re-enabled the sort and filter sidebar on work listings only. Bookmark filtering and sorting is still turned off and will likely be off for a few more days. (The filters are the sidebar that allows you to narrow down a list of works or bookmarks by character, rating, etc.) We will continue to work on the underlying issue. In the meantime, we suggest using the Works Search to help find what you’re looking for.

All works and bookmarks should be showing up normally. Work re-indexing is complete, so we hope to be able to turn on filtering for works again in the next day or two.

Bookmark re-indexing is still ongoing, so it will be several days before we can turn bookmark filtering back on.

Please follow the @AO3_Status Twitter feed or check back here for further updates.

Update 2 Dec: Listings for works, bookmarks, tags, and pseuds are unavailable due to issues with our search index. Our coding and systems volunteers are currently looking into it, and we will keep you updated on our progress. Our Support team is working on a back log, so there might be delays in getting back to users individually. Please consider checking the @AO3_Status Twitter feed or our banner alerts instead.

Update 30 Nov: All bookmarks have been re-indexed and should show up correctly again. Any issues that might still be lingering will be sorted out when we upgrade Elasticsearch, which we're planning for mid-December. Downloads should be working without the need for any workarounds now. Thank you for your patience!

The Good

We recently deployed new code, which fixed a couple of very old bugs and introduced improvements to the kudos feature. Behind the scenes, we've been working on setting up new servers and tweaking server settings to make everything run a little more smoothly during peak times. The end of the year (holiday season in many parts of the world) usually means more people with more free time to participate in more challenges, read more fic, or post more fanart, resulting in more site usage.

One way to measure site usage is looking at page views. This number tells us how many pages (a single work, a list of search results, a set of bookmarks in a collection, a user profile, etc. etc.) were served to users during a certain time frame. Some of these pages can contain a lot of information that has to be retrieved from the database - and a lot of information being retrieved from the database at the same time can result in site slowness and server woes. During the first week of January we had 27.6 million page views. As of November 17 we registered 42.9 million page views for the preceeding week.

We've watched our traffic stats grow dramatically over the years, and we've been doing our best to keep up with our users! Buying and installing more servers is one part of the solution, and we can't thank our all-volunteer Systems team enough for all their hard work behind the scenes. On the other hand, our code needs to be constantly reviewed and updated to match new demands.

Writing code that "scales" - that works well even as the site grows - is a complicated and neverending task that requires a thorough understanding of how all parts of the Archive work together, not just right now, but in six months, or a year, or two years. As we're all volunteers who work on the Archive in our free time (or during lunch breaks), and there are only a handful of us with the experience to really dig deep into the code, this is less straightforward than a server acquisition and will take a little more time.

The Bad

As such, we've been battling some site slowness, sudden downtimes (thankfully brief due to our awesome Systems team) and an uptick in error pages. We can only ask for your patience as we investigate likely causes and discuss possible fixes.

For the time being, we have asked our intrepid tag wranglers to refrain from wrangling on Sundays, as this is our busiest day and moving a lot of tags around sadly adds to the strain on the current servers. We sincerely apologize to all wrangling volunteers who have to catch up with new tags on Monday, and to users who might notice delays (e.g. a new fandom tag that's not marked as canonical right away). From what we've seen so far, this move has helped in keeping the site stable on weekends.

The Ugly

We are aware of an issue with seemingly "vanishing" bookmarks, in which the correct number of bookmarks is displayed in the sidebar, but not all are actually shown. The most likely culprit is our search index, powered by a framework called elasticsearch. All our information (work content, tags, bookmarks, users, kudos, etc. etc.) is stored in a database, and elasticsearch provides a quicker, neater access to some of this data. This allows for fast searches, and lets us build lists of works and bookmarks (e.g. by tag) without having to ask the database to give us every single scrap of info each time.

It appears now that elasticsearch has become slightly out of sync with the database. We are looking into possible fixes and are planning an elasticsearch software upgrade; however, we must carefully test it first to assure data safety.

This problem also affects bookmark sorting, which has been broken for several weeks now. We are very sorry! If you want to know if a particular work has been updated, please consider subscribing to the work (look for the "Subscribe" button at the top of the page). This will send you a notification when a new chapter has been posted.

(Note: Since we're sending out a lot of notifications about kudos, comments and subscriptions every day, some email providers are shoving our messages into the junk folder, or outright deny them passage to your account. Please add our email address to your contacts, create a filter to never send our emails to spam, or check the new "Social" tab in Gmail if you're waiting for notifications.)

A problem with file downloads only cropped up fairly recently. We don't think this is related to the most recent deploy, and will investigate possible causes. In the meantime, if a .pdf or .mobi file gives you an error 500, try downloading the HTML version first, then give it another shot. This should help until we've fixed the underlying problem.

What You Can Do

If you have not already done so, consider subscribing to our twitter feed @AO3_Status or following us on Tumblr. You can also visit the AO3 News page for updates in the coming weeks or subscribe to the feed.

We thank everyone who has written in about their experiences, and will keep you all updated on our progress. Thank you for your patience as we work on this!


Pages Navigation