AO3 News

Post Header

2011-01-07 17:33:17 -0500

Welcome to Revision 3520, up from 3483. This is actually a few different releases bundled together, because we had a little flurry of activity over the holiday period which we didn't post release notes for.

Most Archive users will have noticed that the Archive was experiencing very heavy loads during the holiday period, as Yuletide and other holiday challenges brought high traffic to the site. We know that our sad little 502 page was showing hir face more than any of us would have liked! However, our coding and Systems teams deserve a MASSIVE shoutout for their tireless work keeping the site up and dealing with the performance issues. They worked right through the holiday to keep pushing code improvements and tweaking server setups. We'd especially like to thank sysadmin Sidra for the incredible amount of effort she put into coaxing our poor beleaguered servers into working: her work helped get the number of 502s down from over 9000 (on 25 December) to just 9 a few days later. Go Sidra!

If you're wondering whether the Archive will be this slow NEXT year, then we hope the answer is no! The stress test this year taught us some more about how to configure things for the best possible performance. Even better, we're about to buy some new servers which our hosting service described as 'beasts'. More news on that coming soon...


Static pages for collections

As part of our performance enhancements, the wonderful Elz worked till the eleventh hour to put in static pages for collections (you may have seen us mention these a time or three on @AO3_Status). These are pages where the whole page is created once and then saved, so it is served up quickly for the next user. They can't have any dynamic information (comment counts, hit counts), but they are MUCH faster at peak times.

All collections have static pages - the url is They generate when people access the urls, so they are only noticeably faster for heavily visited collections. Check out a few of the following:

Share via Twitter (or not!)

We've noticed lots of our users sharing recs for AO3 works via Twitter. We thought you might appreciate a quicker way of doing this, so we've added a Twitter button which fills in the title and url for you! Just click the 'Share' button to see the button.

We know not all our users will be comfortable with a feature that makes it really easy to share links on external services, so we've included an opt-out - just check out your preferences to select this. If you opt out, it will disable the Twitter option on your works now, and if we introduce sharing options for other external services in the future it will also disable those. (However, opting out won't actually prevent people from sharing links to your work anywhere, it will just make them do more of the work themselves. If you want a higher level of control over who sees your work, you can restrict your works to Archive users only.)

Improvements to kudos

Since we introduced the 'kudos' option a couple of weeks ago, over 51,000 kudos have been left and we've had a ton of great feedback about it. We've fixed a few things that were not quite right with the feature: it's now 'kudos' everywhere (no weird singular usage) and you can opt out of kudos notifications without turning off your comment notifications.

Known Issues

See our Known Issues page.

Release details


  • Added server-relieving caching options to collections and challenges
    • Added static pages for collections (fandom pages, work pages, etc).
    • Added ability to leave kudos from static work pages
    • Ensured that static pages would persist across deploys.
  • Added 'Share on Twitter' button
  • Added the ability to opt-out of allowing sharing of your works on external services (e.g. Twitter).
  • Changed kudos-related wording in notification emails and on work pages to take care of the singular/plural issue - no more strange singular 'kudo'!
  • Removed revision number from page titles
  • More automated tests, continuing our mission to test ALL THE THINGS.

Bug fixes

  • Removed author names from "share" box for anonymous works (whups)
  • Fixed byline bug causing names of anonymous authors to appear in static files and downloads
  • Made it possible to edit a work in a closed challenge without being kicked out of the collection
  • Fixed collection links in anonymous work blurbs showing up as unparsed HTML code
  • Fixed a bug where you couldn't edit tags on a work if they were related to a meta tag
  • tweaked collections pages for display in Internet Explorer
  • Cleaned up story notes section, fixing issues which were causing links to endnotes to be encased in invalid HTML and thus look weird.

Systems changes

  • Added option to turn off kudos notifications
  • Change thinking sphinx delta delayed jobs into an hourly cronjob (relieved load on the server)
  • Changing reindex to every two hours (relieved load on the server)
  • Gave delayed job its own restart
  • Made it possible to run deploy quick on otw2 server (saves some work during repeated deploys)
  • Updating static sweeper


Post Header

2011-01-02 17:04:09 -0500

Do you love the Archive? Do you have skills in database analysis? Would you like to make these great two tastes taste great together?

We're looking for a MySQL database developer who can help us optimise our database performance on the Archive of Our Own. The Archive is run on MySQL 5.0 and Rails 3, but you don't need to know Rails to help -- we would really like someone who can look over our database query logs and help us spot inefficiencies!

You'd be joining a team of dedicated and passionate volunteers with a love of fandom and tech-geekery. We're looking for someone with the time and skills to help us evaluate and optimise our current setup — this can be a short-term commitment, although if you are able to join us for the long haul your skills will be very welcome.

If you're interested, please contact our Volunteers and Recruitment Committee using 'MySQL database developer' as your subject line, and let us know your skills and experience.

Your Archive needs YOU!


Post Header

2010-12-31 20:27:53 -0500

2010 was a wonderful year for the AO3! It was our first full year in Open Beta! We ran two big challenges and lots of smaller ones! We filled up our servers, introduced a bunch of new features, made 1451 code commits and got shout-outs all over the web.

Our team had an eventful year! People moved countries, travelled across continents, changed jobs, got PhDs and had babies. Oh, and we built this great Archive for fandom to enjoy! We feel a great sense of achievement!

We are incredibly grateful to everyone who worked on the Archive during 2010: coders, testers, sysadmins, tag wranglers, support team, designers, and many more!

We're also hugely grateful to all Archive users and members of the OTW for your great feedback, your generous financial contributions (which will pay for beautiful new servers this year), and most importantly, your fannish energy and creations, which are the true heart of the Archive. At the close of 2010 we had:

  • 7,927 fandoms
  • 11,504 users
  • 128,806 works
  • 4,949 series
  • 591 collections
  • 164,450 tags (of which 100,046 are 'canonical' and can be filtered on, 79,294 are character tags, and 45,569 relationship tags)
  • 40,299 kudos

We think this is an AMAZING output for just over a year of real usage! GO GO FANDOM!

We're looking forward to seeing what awesome new fannish output 2011 brings! Thanks everyone for your support! Happy New Year!


Post Header

2010-12-21 16:03:27 -0500

Welcome to revision 3453, up from revision 3435! This release contains some minor bugfixes as well as a few nice feature enhancements! Thanks, as always, to the wonderful coders and testers who contributed to this release.


Kudos improvements!

The new Kudos feature is very popular! However, a number of people found that they had accidentally added Kudos to their own works and felt a little bashful about it. In response to user feedback, you will no longer be able to add Kudos to your own works. Additionally, existing self-applied Kudos have been removed.

In other Kudos news, we've also added notification of Kudos being given for your works. You will receive these notifications so long as you have comment notifications turned on in your Preferences. You will also be able to search for stories by number of Kudos.

Improved Fandom Index Pages!

Fandom index pages have been redesigned with the addition of handy alphabetical jump links to help you find your place in the listing more quickly.

Admin Post Translations

This release also adds an improved system for the translation of Admin Posts, no longer requiring manual linking to the translated versions. Non-English posts will not show up in the RSS feed, but you will be able to subscribe via RSS to posts in the language(s) of your choice.

Non-English admin posts will be automatically linked from the English version, and will show up under the url for the language in question, e.g. the German posts will be at To get the rss feed, simply append .rss to the relevant url:

Note: Change to Search Function

For performance reasons, you will no longer be able to search on the full text of works content. Summaries and notes are still searchable.

Known Issues

See our Known Issues page.

Release details


  • Improvements for fandom index pages (sorted alphabetically, with quick jump links)

  • List of fandoms in collections and challenges can now be filtered by media type

  • Added ability to search for stories by number of kudos

  • Added notifications for kudos (unless you turned off comment notifications under Preferences)

  • Added a fix that will prevent users from adding kudos to their own works

  • Made unposted chapters in a chaptered work more noticable

  • Enabled comment management for admins (mark as spam, delete)

  • Better system for admin post translations (different versions of an admin post don't have to be manually linked anymore)


  • Fix for updating all unposted stories in a challenge as defaulters

  • Fixed a bug causing some users to be without default pseuds; if you found that you got multiple 500 errors when logged in (couldn't view profile, works, or sign up for challenges) then this should have fixed your problem


Post Header

2010-12-15 23:01:21 -0500

Welcome to revision 3435, up from revision 3403! We have a bunch of fun things in this revision along with some much-needed performance enhancements! Thanks to all the wonderful coders and testers who contributed to this release, and a special thanks to mumble, who put the release notes together!


RSS feed for admin posts

If you want to keep track of what we post about the site, you can now follow our admin posts via an RSS feed: add to an RSS reader of your choice.

CSS styling for works!!!!

For a long time our users have been asking for better ways to style their works on the AO3. We really wanted to give users greater control over their font choices, positioning, typography, etc, but accessibility is also a major concern for us, and we wanted to make sure that it would be easy for users to override more complex styling if they wanted or needed to. We’re thrilled to announce that we have now introduced ‘work skins’, which allow us to have our elaborately iced, twelve-tier cake with sparkles on and eat it too! For information on the new feature and how it works, check out our Tutorial on styling works (coming soon!).


Lots of people have asked us for a quick way to leave love or appreciation for a work, and we were very happy to oblige! You can now leave ‘kudos’ at the click of a button – works with kudos will display this happy little AO3 dude: Kudos: Mini AO3 icon with an arc of hearts over its head along with names and numbers of folk who left kudos. We think this will be a great way for people to spread the love during the holiday season, when finding time to read fic in between family commitments can make it hard to leave long comments.


Finally, we couldn’t let this release go by without welcoming the newest addition to the AO3 family! After a month of producing amazing amounts of code, our senior coder Astolat went one better and also produced a beautiful baby girl! Baby Hypatia was born on the 7th December, weighing 7lbs 4oz. Astolat was coding right up until her waters broke, and sending emails about our new servers while she was in labour! (Our staff are crazy dedicated, OMG.) Mommy and baby are doing well – KUDOS ASTOLAT! Kudos: Mini AO3 icon with an arc of hearts over its head

Known Issues

See our Known Issues page.

Release details


  • Added the ability to post works without previewing them first (especially handy for editing!)
  • Added the option to leave ‘kudos’ on works.
  • Made ‘leave comment’ box visible by default.
  • Added new styling option for works via ‘work skins’:
    • Renamed “Light Style” to “Hide/Show Creator’s Style”
    • Renamed “Disable User Generated Styles” to “Always hide custom styles”
    • Created public “Basic Formatting” work skin
  • Made comment form show by default
  • Added additional caching to works and chapters for better performance
  • Comments now display user’s local time instead of server time
  • Entering a wrong collection name in the Post New form now yields an error message
  • More improvements and bugfixes for automated tests
  • added RSS feed for admin posts (
  • Tweaks to the “share” option to spiff it up a bit!


  • Fixed a bug causing 500 errors for some people when posting to fulfil challenge assignments.
  • Unbroke OpenID login
  • The share box is now disabled for users without javascript so that it doesn’t hog screen real estate.
  • Fixed issue causing 500 errors when downloading fics with related works
  • Fixed some overlapping issues in the comments section
  • Fixed some mangled text on the sign-up page


Post Header

2010-12-11 14:51:28 -0500

Es ist wieder jene Zeit des Jahres, in der Fans sich am Geschichtenwichteln beteiligen, Glühwein trinken und das Eigene Archiv verfluchen, weil es wieder mal so verdammt langsam ist. Denn nach einem anstrengenden Tag in der Innenstadt, in der man sich durch Menschenmassen drängeln und an der Kasse ewig warten musste, gibt es nichts Nervigeres als mit der Fehlermeldung Error 502: Page did not respond in a timely fashion konfrontiert zu werden, wenn man zum Entspannen auf seine liebste Fanfiction klickt.

Bitte glaubt uns: Wir wissen, wie das ist. Wir sind auch Fans und benutzen das Eigene Archiv, und nach einem Tag voller Schlangestehen oder dem Programmieren einer neuen Funktion stört uns auch nichts mehr als ein Fehler 502, wenn wir zum Entspannen auf unsere liebste Geschichte klicken.

Deshalb möchten wir euch wissen lassen, dass wir etwas dagegen tun. Wir arbeiten gerade an einem neuen Systemaufbau (dank eurer Unterstützung mitsamt schicker neuer Server) der die Leistung deutlich verbessern wird.

Wir haben eine Menge Vermutungen angestellt, bevor wir die letzte große Aktualisierung durchgeführt haben. Ein wichtiger Teil dieses Updates war die Einführung von Rails 3, das die Grundlage für die Umgestaltung des Systemaufbaus bildet. Sowohl währenddessen als auch kurz danach haben wir schon einige Leistungsverbesserungen auf den Weg gebracht, aber letztendlich gibt es leider keinen Ersatz für die Erkenntnisse, die wir aus den tatsächlichen Nutzungsdaten des Archivs gewinnen.

Wir haben nun 11.000 Nutzer und Nutzerinnen und 120.000 Werke und sind damit wesentlich schneller gewachsen, als wir uns jemals erträumt haben.

Was genau wird sich durch den neuen Aufbau verändern?

Derzeit läuft das Archiv auf zwei Servern – einer für die Datenbank, der andere für die Rails-Anwendungen.

Die neue Systemarchitektur wird in etwa so aussehen:

Diagram der Servertopologie

Zum Vergrößern auf die Grafik klicken

Kurz gesagt werden wir auf einen Systemaufbau umsteigen, der es uns erlaubt, in der Zukunft leicht zusätzliche Hardware zur Leistungsverbesserung anzuschließen.

Wir kaufen gerade die neuen Server für diesen Aufbau, aber im Moment der Zeitpunkt für eine so drastische Veränderung ungünstig. Der Umstieg auf einen neuen Systemaufbau ist keine triviale Sache, und wir verschieben ihn deshalb lieber aufs neue Jahr, wenn wir mehr Zeit für Ausfälle haben und nicht mehr so viel los ist. Im Eigenen Archiv herrscht derzeit reger Betrieb (nicht nur durch Yuletide, auch wegen anderer Wichtelaktionen) und wir wollen nicht riskieren, dass uns das alles um die Ohren fliegt.

Daneben ist Yuletide, unser erster und größter Testfall, in diesem Fall Teil einer langfristigen Lösung. Tausende Nutzer und Nutzerinnen, die alle zur selben Zeit auf dieselben Teile des Archivs zugreifen wollen, werden unsere Systeme bis zum Anschlag belasten und uns helfen herauszufinden, wo die größten Engpässe liegen. Das wiederum zeigt uns, welche Prioritäten wir beim Kauf der Hardware setzen müssen.

Das heißt aber auch, dass im Moment alles etwas langsam läuft. Das tut uns sehr leid. Auch aus diesem Grund ist das AO3 immer noch in der Betaphase. Wir werden das “Beta”-Schild auch nicht herunternehmen, solange wir nicht einigermaßen sicher sind, dass das System mit möglicherweise Millionen von Nutzern und Nutzerinnen und Geschichten umgehen kann, ohne dass uns alles um die Ohren fliegt oder nur im Schneckentempo läuft. Und soweit sind wir noch nicht.

Wir müssen euch also um Geduld bitten. Yuletide und andere Geschichten-Tauschaktionen – und die daraus resultierende intensive Nutzung – sind für das AO3 und diejenigen, die es programmieren und verwalten, auf lange Sicht sehr wertvoll. Es ist alles Teil des Betaprozesses – habt also bitte Verständnis dafür, dass solche Hänger nötig sind, um das Eigene Archiv langfristig voranzubringen.


Post Header

2010-12-11 14:51:22 -0500

Esta es la época del año cuando los fans participan en en intercambios de fics, beben sidra fermentada y maldicen a Un Archivo Propio por ser tan @#$!% lento. Porque después de un día de lidiar con multitudes o esperar en las cajas de las tiendas departamentales, nada te frustra más que ver un Error 502: Page did not respond in a timely fashion en tu fic favorito para confortarte.

¡Creénos, lo sabemos! También somos fans y usamos el AO3, y después de un largo día luchando contra las multitudes festivas o programando, nada nos frustra más que nos salga un Error 502 en nuestro fic favorito de confort, tampoco.

¡Queremos que sepas que estamos haciendo trabajando en esto! Actualmente estamos en el proceso de armar una nueva arquitectura de sistema — incluyendo la inminente adquisición de nuevos y brillantes servidores, gracias a tu ayuda — la que ayudará a mejorar nuestro desempeño de forma considerable.

Podemos (y lo hemos hecho) hacer montones de presunciones al realizar nuestro último lanzamiento, el cual incluyó la actualización a Rails 3 que era necesaria antes de proceder con la remodelación de los sistemas, y dimos lo mejor para mejorar el desempeño tanto en el proceso como inmediatamente después del lanzamiento. Pero al final, no hay substituto a observar resultados en el uso real cotidiano del sistema.

En este momento contamos con 11.000 usuarios y 120.000 trabajos hospedados: ¡estamos creciendo más rápido de lo que nunca habíamos soñado!

¿Qué es lo que va a suceder con este nuevo diseño?

Actualmente estamos trabajando con dos servidores — un servidor para una base de datos individual y otro servidor para la aplicación Rails.

La nueva arquitectura se verá más o menos así:

diagram of server topology

Haz click en la imagen para ampliar

Básicamente, estamos moviéndonos a un diseño donde podríamos, en un futuro, fácilmente conectar máquinas adicionales cuando se requieran, para mejorar el desempeño.

En este momento estamos comprando los nuevos servidores para implementar esta nueva arquitectura, pero no creemos que sea una buena idea implementar un cambio de sistema tan grande como este en este preciso momento. La implementación de esta nueva arquitectura no es trivial y planeamos hacerla en una forma más controlada y con mucho más espacio para tiempo fuera de línea cuando las cosas se calmen entrando el año. Están sucediendo muchas cosas en el AO3 – no sólo Yuletide, sino montones de otros intercambios festivos de fic – y realmente no queremos que todo nos explote.

Pero también, en este proceso, Yuletide – nuestro principal y más grande caso de prueba – es parte de una solución a largo plazo. Tener miles de usuarios, todos golpeando a las mismas secciones del archivo al mismo tiempo y estirando nuestros sistemas al límite, nos ayudará a identificar donde están los cuellos de botella más grandes y eso nos dirá cuales deberán ser nuestras prioridades al comprar el nuevo hardware.

Esto significa que las cosas están lentas ahora. Y estamos muy apenados por eso. Es parte de las razones por las que AO3 aún está en proceso de prueba o beta. No quitaremos el aviso de “Beta” hasta que no estemos razonablemente confiados de que el sistema puede manejar millones de usuarios e historias, sin caerse, explotar o tener grandes estancamientos. ¡Aún no llegamos a eso!

Así que les pedimos paciencia. Yuletide y otros intercambios – y el resultante uso intenso – son tremendamente útiles a largo plazo para AO3 y sus diseñadores y administradores de sistema. Es parte del proceso de prueba – ¡por favor entiendan que todo esto es necesario para que nosotros llevemos a AO3 a donde necesita estar!


Post Header

2010-12-11 14:27:47 -0500

This post is also available in Deutsch and Español.

This is the time of year when fans participate in fic exchanges and drink mulled cider and curse out the Archive of Our Own for being so @#$!% slow. Because after a day of fighting holiday crowds or waiting on lines in department stores, nothing bums you out like getting Error 502: Page did not respond in a timely fashion on your favorite comfort fic.

Believe us: we know!  We're fans too, and we use the AO3, and after a long day fighting holiday crowds or coding a new feature, nothing bums us out like getting Error 502 on our favorite comfort fic, either.

We want you to know we're doing something about it! We're currently in the process of putting together a new systems architecture -- including the imminent purchase of several very shiny new servers thanks to your support -- that will boost performance substantially.

We could (and did) make lots of guesses going in to our latest big deploy, which included the Rails 3 upgrade we needed to make before we proceeded with the systems revamp, and we did our best to put in performance improvements both on the way and immediately after the deploy. But there is no substitute in the end for looking at actual results from real-world usage.

We're now at 11,000 users and 120,000 works: we've grown much faster than any of us ever dreamed!

So what exactly is going to happen with this new design?

We currently are running on two servers -- a single database server and a single Rails app server.

The new architecture will look something like this:

diagram of server topology

Click image to see larger

Basically, we are moving to a design where we can easily in future plug in additional machines as needed for performance improvements.

We are currently buying the new servers to implement this architecture, but we don't think it's a really good idea to implement a major system change like this right now. Implementing this new architecture isn't trivial, and we plan to do it in a more controlled way and with a little more leeway for downtime once things slow down in the new year. There's a lot going on at the AO3 - not just Yuletide, but loads of other holiday fic exchanges - and we don't really want the whole thing to go boom.

But also, in this process, Yuletide, our first and biggest test case, is part of the long-term solution. Having thousands of users all banging on the same parts of the archive all at once and stressing our system to the limits helps us identify where the biggest bottlenecks are, and that in turn tells us what our priorities need to be when we buy our hardware.

That means things are slow now. And we're very sorry about that. It's part of why the AO3 is still in beta. We won't be taking the "beta" sign off until we have reasonable confidence that the system can handle potentially millions of users and stories, without falling down and going boom or having major slowdowns. But we are not there yet!

So we need to ask for your patience. Yuletide and other exchanges - and the resulting intense usage - are tremendously useful in the long run for the AO3 and its designers and sysadmins. It is all part of the beta process - so please understand it is all necessary for us to get the AO3 to where it needs to be!


Pages Navigation