January 2013 - our server setup
Over the course of 2012, the demands on all our sites grew. In particular, the number of users accessing the Archive of Our Own each month dramatically expanded from 963,818 in January 2012 to 2,970,103 by December 2012. This demanded a significant expansion of our hardware: we bought 3 new servers and were lucky enough to have another 2 servers donated.
As of January 2013, the OTW owns 11 servers and one switch to communicate between them, and pays for space on 3 virtual servers (some of these will be decommissioned in the coming months).
Our physical servers are at colocation facilities. Our current hosting costs for our physical machines amount to $1,640 a month, and we pay another $370 a month for virtual hosting.
Planning for expansion
Systems have spent a lot of time thinking about how to manage the current demand and plan for continuning expansion going forward. We have put in processes which will allow us to add servers to the organization with efficiency so that additional growth can be done with far less systems administration work that has been required in the past.
One of the issues we have had in the past is that the systems have been maintained individually. When the number of servers is small, this can be maintained but it becomes unwieldy as more servers are obtained. We have been trying to balance looking after the servers with committing to the work needed to automate the installation and configuration of the systems needed to provide the Archive and Fanlore and the other org sites.
Over November and December 2012 the group spent around around 25 hours a week automating the installation (with fai, the fully automated install system) and configuration (with cfengine3, ). Once this work is complete we should be able to provision new servers both physical and virtual quickly and consistently.
As of January 2013, the machine specifications and jobs were as follows:
|otw-admin (was OTW1)||ProLiant DL360 G5, E5420@2.50GHz with 16GBytes of Ram and 140GB of RAID 10 disc.||OTW Tools Administration host for the OTW. Hosts cf-engine3 (see below) servers, dhcp and tftp. Redis slave for the Archive, Mysql server for internal databases, local Debian Linux repository, xtrabackup manager (Mysql backup).|
|otw-gen01 (was OTW2)||ProLiant DL360 G5, E5420@2.50GHz with 16GBytes of Ram and 140GB of RAID 10 disc.||OTW Projects New installation currently under testing. Will host Fanlore, Transformative Works and Cultures (the OTW journal) and Symposium (TWC’s blog). Uses Squid, Apache, memcached, MySQL.|
|OTW3||Supermicro X8DTU, 24 gig of RAM, dual E5620 4 cores @ 2.40GHz, 4*143GB SAS discs||Archive of Our Own Runs nginx and Squid to provide the front end of the Archive. Runs the following sets of unicorns: 5 unicorns for web spiders, 5 unicorns for comments, kudos and adding content,16 unicorns for retrieving works or their comments and 30 general purpose unicorns.|
|OTW4||Supermicro X8DTU, 24 gig of RAM, dual E5620 4 cores @ 2.40GHz, 4*143GB SAS discs||Archive of Our Own Runs the following sets of unicorns: 18 unicorns for comments, kudos and adding content,10 unicorns for retrieving works or their comments and 50 general purpose unicorns. Resque workers (these run jobs that do not need to be done immediately, such as sending email).|
|OTW5||Supermicro X8DTU, 48 gig of RAM, dual X5650 6 cores @ 2.67GHz, dual intel X25 80GB disc||Archive of Our Own Mysql primary (database server for storing works in the Archive). Memcached (used to speed up the archive). Redis (used to store data that needs to be stored quickly, such as page hits, etc).|
|Qnap||QNAP TS-809U, 8*2GB discs.||Archive of Our Own Storage device. Used for backups, work downloads, and shared binaries. Hosts fai ( fully automated install system, http://fai-project.org )|
|switch||16 port netgear dumb switch.||Networking switch. Used for communications between internal servers.|
|tao||Virtual machine with 1GB of RAM||OTW Tools Service machine: primary email support, Mailman (mailing lists), DNS hosting, other support services and tasks|
|zen||Virtual machine with 1GB of RAM||OTW Projects Currently used to host Fanlore, Transformative Works and Cultures (the OTW journal) and Symposium (TWC’s blog). Soon to be decommissioned|
|buddha||Decommissioned.||All sites (transformativeworks.org, elections site and opendoors) have been moved to a third party Drupal vendor.|
|stage||Virtual machine with 1.5GB of RAM||Test server. Once the new versions of Fanlore and our other sites go live, the test sites will go to the new physical stage hosted in a separate colocation facility.|
|stage (2nd site)||HP ProLiant DL385 G2 2*Dual-Core AMD Opteron(tm) Processor 2218, 32GB of RAM, 1TB of raid 6 disc over 8 discs.||Test server. Used to host the Archive software before it goes live. Mysql secondary for all of the org’s mysql servers. Secondary for redis for the archive.|
|dev (2nd site)||HP ProLiant DL385 G2 2*Dual-Core AMD Opteron(tm) Processor 2218, 32GB of RAM, 1TB of raid 6 disc over 8 discs.||Development server. Used to provide a Unix environment for Archive developers, so people don’t need to set up the code on their own machines to code for us.|
|spine||Virtual machine with 2 GB of RAM||Service machine. Used to host offsite backups and other services.|
|ao3-db01 New!||Supermicro SuperServer SYS-6027R-TRF 8*cores @ 2.6GHz, with 256 GB of RAM, RAID controller with battery backed up cache, 2* 2TB SATA discs Intel 910 SSD 800GB PCIE||The new database server. We have spent a considerable amount of money here both on RAM and the intel 910 PCIe storage. This system should be able to support the Archive for a reasonable amount of time. If we need more performance we will have to buy additional machines and shift memcached and redis to other machines, later buying a replacement machine with even more memory. We are willing to spend so much money on RAM as our developer resources (the number of hours people can devote to coding the Archive) are even more tightly constrained than our financial resources.|
|ao3-app01 New!||CSE-815TQ-R700WB server, E5-2670 8*cores @ 2.6GHz, with 256 GB of RAM, RAID controller with battery backed up cache, 2* 2TB SATA discs||These machines will just run the Archive application. If there is significant spare RAM then we will run memcached instances and add them to the memcached cluster.|
|ao3-app02 New!||CSE-815TQ-R700WB server, E5-2670 8*cores @ 2.6GHz, with 256 GB of RAM, RAID controller with battery backed up cache, 2* 2TB SATA discs||Same as ao3-app01|
Virtual machine: a server that looks like an actual computer but is actually software built on top of a larger, higher performance server. Virtual machines are ideal for web servers and other basic workhorse systems.
Storage device: A system that is mostly disk space and networking. Imagine a gigantic external hard disk times a billion.
Service machine: A system that runs mostly behind the scenes programs that Joe User never sees, but OTW staff may need.
KVM: Keyboard/Video/Mouse: Servers generally do not come with these, but are just big boxes full of disks, memory, CPU, and lots and lots of fans. To talk to a server directly, while in front of it, you generally need a KVM.
DNS: Dynamic Name Service. What tells other computers (like yours) where to find sites like archiveofourown.org.
Colocation: Remote hosting site where servers are kept. Hosting costs include power, cooling, and someone to physically work with the machine when needed.
The new servers have a total of 48 cores of compute, 768GB of RAM, and 800GB of very fast storage. We are currently providing the Archive on 28 cores of compute and 192GB of RAM.
Archive of Our Own - server setup
As of January 2013, buying new servers had allowed us to significantly restructure the systems architecture for the Archive of Our Own. The technically minded can see the basics of our setup below (this is a simplified version of the current configuration with systems such as the mail server and the secondary mysql servers removed: