AO3 News

Post Header

Published:
2019-02-09 17:01:58 -0500
Tags:

Soon we'll be replacing our current method for generating work downloads with the Calibre command line interface. Our current patchwork system of third-party tools and custom code comes with a number of bugs and limitations, and moving to a well-maintained and documented application like Calibre will enable us to better maintain and even enhance the download feature going forward.

Because this is an entirely new system, there will be some differences between old and new files. We've provided an overview of these changes and some previews below, but we're still testing these changes, so there may be some minor differences between the previews and the final versions. Additionally, exactly how a file looks and behaves may vary depending on your device, software version, and settings.

HTML & PDF

HTML files serve as the basis for all of our download files, so it was necessary to make a few tweaks to help Calibre create tables of contents for other download types. Most of these tweaks are invisible, but we did make one visible change: the work title is now included at the top of the file, above the URL. Aside from these minor changes, HTML downloads will remain the same.

Because generating PDF downloads with Calibre would require us to install additional software on our servers, we will continue to use wkhtmltopdf to create PDFs, although we have upgraded it to a newer version. Like HTML downloads, PDFs will now include the title at the top of the first page but otherwise remain the same.

EPUB & MOBI

The biggest changes will be to MOBI and EPUB files, which are currently very different from each other due to limitations that existed when our system was created nearly ten years ago. The new code will give these files a more consistent look and feel and allow readers to take advantage of some features ebook readers provide.

EPUB

A current EPUB open in the iPhone Books app. The work meta is displayed in two columns inside a box. A new EPUB open in the iPhone Books app. The work meta is displayed as a bold heading, e.g. Rating, above the relevant tags.

The new EPUB files will do away with the two-column display of work meta (i.e. tags and statistics), which doesn't work well for small screens, lots of tags, or large font sizes.

This update will also fix the bug where EPUB files were missing chapter endnotes.

You can download an example of the new EPUB files to preview on your device.

MOBI

First page of a current MOBI file on the iPhone Kindle app, cropped to show the table of contents links provided at the top. A new MOBI file on the iPhone Kindle app, cropped to show the file's table of contents in the app menu.

Our current system provides a list of links on the first page as a makeshift table of contents, but Calibre will create files designed to work with your device or app's built-in table of contents function. (As you navigate with the shiny new table of contents, you may also notice each chapter begins on a fresh page.)

The text of a chapter in a current MOBI file on the iPhone Kindle app. The text of a chapter in a new MOBI file on the iPhone Kindle app.

Calibre will also add margins to prevent the text from running against the edge of the page. Additionally, we've switched from using indented book-style paragraphs to using vertical margins to create blank lines between paragraphs, bringing MOBI files in line with how text is formatted on the site and in other download files.

This update also brings a major improvement for internationalization: MOBI files of works written in non-Latin alphabets will no longer be automatically transliterated! Please note, however, that MOBI offers only limited support of right-to-left languages like Hebrew and Arabic, so Kindle users will need to use AZW3 to read works in those languages.

You can download an example of the new MOBI files to preview on your device.

AZW3

We'll also be adding a new download format: AZW3, which was created by Amazon for use with Kindle devices and apps. It's a newer format than MOBI, and it will allow us to offer Kindle users downloads of works in right-to-left languages, but it will otherwise look the same.

AZW3 downloads will be available on all works, but if you rely on Send to Kindle to transfer files to your device, you'll need to stick with a different format: Amazon doesn't allow you to send this file type. However, you can still transfer AZW3 files to your Kindle by downloading them to your computer and connecting your device.

If you're using the Kindle app on an Android device, you can move AZW3 files you've downloaded to your SD card's Kindle folder in order to access them. Unfortunately, there doesn't appear to be a workaround for the iOS Kindle app, so iPhone and iPad users will need to use another download format or a different, non-Kindle app.

Deploy date

While this upgrade won't immediately resolve all of the existing issues with downloads, it's an important part of the process, and it will ultimately make the Archive more secure and more maintainable. (After the deploy, our release notes will have more detail on specific bugs this change has fixed.)

We're still in the testing stages, but we plan to deploy these changes sometime after February 11. When the time comes, we'll announce it on our @AO3_Status Twitter account.

No downtime is expected, but if you have a work already loaded in your browser at the time of the update, you'll need to refresh the page to download it.