How to build an 18W, 4 terabyte, commodity x86 Linux server

Image links to Amazon

Can you build a Linux server for web, email, printing, and 4 terabytes of media serving purposes from commodity x86 parts, and come in under 20W? Absolutely!

If you run a server at home 24/7, and it uses, say, 150W, that’s 1.3 MWh (megawatt-hours) of electricity per year, and probably $130 or so of utility cost.  You can do better!

I recently assembled a server with the following specs:

(note, some Amazon product links follow):

Out of the box, the Kill A Watt [amzn] showed about 20W.  After running this script to further push down some power consumption, it came in at 18W:

#!/bin/bash

# Idle media disks
# (should set shortish spindown time too...)
hdparm -y /dev/sdb
hdparm -y /dev/sdc

# 100baseT saves power over GigE
ethtool -s eth0 autoneg off speed 100
ethtool -s eth1 autoneg off speed 100

# kill Wake On Lan
ethtool -s eth0 wol d
ethtool -s eth1 wol d

# set ALPM on SATA links
for FILE in /sys/class/scsi_host/host?/link_power_management_policy; do
    echo min_power > $FILE
done

# Set PCI Express power management
echo powersave > /sys/module/pcie_aspm/parameters/policy

# Try to keep 1 core idle when possible
echo 2 > /sys/devices/system/cpu/sched_smt_power_savings
# Disable the nmi watchdog
echo 0 > /proc/sys/kernel/nmi_watchdog

Ok, idling the 4T of storage might be cheating a little, but in reality, they hold music and movies, and simply do not need to be spinning all the time.  (Without these drives in the system, it comes in at about 16W.)

At 18W, that’s 158 kWh/year, or about $16.  An 88% reduction!  And, the utility savings would almost pay for the motherboard!

I don’t know if non-mobile Atoms would fare as well; my original test of an Atom motherboard was lackluster in part due to lack of power management, but then I didn’t have the efficient power supply either, so I’m uncertain.  I’d imagine that the new AMD E-350 (Brazos) motherboards would do very well too, but I’ve not been able to test one out.

And, one more bit of full disclosure – sandeen.net isn’t running on this box yet, the migration isn’t complete.  Still, quick tests of WordPress running on the platform don’t seem any slower than the current old AMD box (a 40W hog ;) ).  And if LKML keeps spamassassin really busy, maybe it’ll inch up in power a bit.  I’ll let you know when you’re reading this blog on the new machine, and what the average wattage draw is over a week or so.

So, for you geeks out there with home servers, how much power does it draw?  Would you consider something like this for your next build?

55 thoughts on “How to build an 18W, 4 terabyte, commodity x86 Linux server

  1. I wonder if my UPS built-in load meter reads out low (I’ll have to put it on the kill a watt to check). But I have a reading around 40w with two servers running – one is a dual-core conroe 6600 (2.4 ghz) and the other is a new sandy bridge i5-2500k. Between them they have 7 spinning drives of 1.5 or 2T each and the sandy has two 40gb ssd’s. There’s 6gb of ram in the conroe and 8 gb in the sandy.

    And my ups shows load at 75w with the 24″ panel turned on, and it bounces around 38-42w with the panel turned off. Of course neither box is doing anything right now and both systems are speedstepped down to 1.6 GHz. The power supplies are premium but nothing too green.

  2. I’ve been running a via epia based server for a few years, comes in at around 50 watts with the media drives spun down.

    I’d recently been looking around for a mobile atom board with 4 sata ports, but had somehow missed this one. Thanks for the pointer!

  3. What do you do for redundancy? I have to admit I’m running server class hardware (Dell Poweredge 2400) but only for the hardware RAID. I have recently replaced a larger number of smaller disks with lesser numbers of large disks just to cut power usage (and I cut by almost 50% according to kill-a-watt) but I’ve lost data before and have found redundancy to be more effective in that regard to backups. My luck with backups is horrible. If I loose my wife’s email and file archive again…

    • I will actually have two 2.5″ drives in a mirror for the OS and mail/web data. The 2x 2T drives are for media, and they used to be in a mirror, but I think I will split that and rsync nightly to keep them in sync. For music & movies, a short downtime from one disk going bad is better than a filesystem problem eating both at once, I think.

  4. Rather than the Jetway which wants ATX power and needs the Pico-PSU, might I suggest Intel’s D945GSEJT, which takes 12V DC directly? I suspect it also requires less power.

    It has only 2 SATA connectors, so you’d have to put the root filesystem on one of the 2TB drives, but still.

    • I’m surprised there aren’t more of these boards with direct 12VDC inputs, it seems quite logical at this point. The PicoPSUs are still odd beasts, but even “small” ATX powersupplies are grossly oversized at this point…

      But yes, the limited SATA connectors would be an issue for my usecase, I really wanted 4. I suppose you could always add a PCIE sata card too, but I wonder how many drives the motherboard can supply power to?

  5. Disclaimer: I work with several ARM vendors as part of linaro.org

    I always feel a bit frustrated when I read articles about low power servers @ 20W and there is no mention of ARM-based devices. It reminds me of our failing as part of the ARM community.

    Here are a few random links from googling. [1] talks about the more recent boards (still under 5W) and [2] about a seriously underpowered one by today’s standards that still seem to be enough for some people.

    One thing I’ll readily admit is disk performance or lack there of with these boards since they only take USB hard drives. If that isn’t a constraint, then you should definitely look at these.

    [1] http://jeffbastian.blogspot.com/2011/04/my-pandaboard-has-arrived-it-was-only.html
    [2] http://nslu2watt.dyndns.org/

    • Hi Amit – I know ARM would be lower power, and I would love to run on ARM. Really! I have a sheevaplug, and it’s fantastic at ~6W, but it can only do so much without more connectivity.

      You’re right – I can’t add storage to the ARM solutions I see, other than via external USB disks, and the accompanying performance issues, tangles of cables and less-efficient wall-warts.

      Why are there no mini-ITX formfactor ARM boards with 4 SATA connectors? Get one of those out there for under $200, and you’d have a winner, I think.

        • > 4 SATA connectors! You do know how to inflict pain. :)

          Dumb, but honest question: Why is that so painful? By the time you have one, isn’t at least the second “almost free?” :)

          nb: that board is not available now, and when it is, it’ll be $394. (Edit: oops, bare board is $199; but, still sold out) Sorry, that just doesn’t work for my server needs … But I want to believe! I really do hope ARM becomes useful in this space.

          • We’ve been asking for SATA for years, hence the pain. And after so many years we’ve finally managed to get 1 SATA connector.

            But ARM is starting to look at the server space now, so things should change rapidly. (Fingers crossed)

          • BTW that board does offer an 802.11n option but if you added a USB to ethernet dongle so you have a second serial port and their wifi adaptor or a USB wifi adaptor you could make this into a SAN, Firewall, Router, WiFi router as we.
            With all the serial ports that it offers you could also add a Zig Bee radio for some remote data collection projects.

    • The problem I have with the ARM SoC based “small PCs” is that a lot of the time the SATA ports are connected internally to the USB… at least that’s the case with the Dreamplug. Other than that I’m very happy with my Dreamplug.

  6. Pingback: Eternal Vigilance is the Price of Efficiency, part 1 | Eric's Blog

  7. Pingback: What’s up with the Intel Atom? | Eric's Blog

  8. Pingback: Quora

  9. Great article!

    Are you using the case fan with this board? And does it ever completely turn off with the bios smart fan configuration? I’m thinking of buying this board, but I wouldn’t want to hear any fan noise if the system is completely idle, while at the same time it seems that a case fan is required when the system is under high load.

    • The Lian Li case I got did come with a large fan on the front. To be honest I have not paid attention to the fan; I’ll try to look at the bios settings soon and reply again with more info. Fiddling with the sensors infrastructure might also be useful there.

      The board is a bit pricey still; I’d be interested to know how an AMD Brazos board would fare power-wise; they are available at decent prices now.

      • FWIW f71882fg.ko (F71882FG Hardware Monitoring Driver) handles the fan sensors on the board; as it sits idle right now the main fan is at 675RPM. I think the smart fan control is on in the BIOS.

        • And at what temperature does it idle? I wonder whether it would be possible to configure fan control so that the fan is off when idling.

          By the way, I read some problems about the GMA3150 running hotter on linux than on windows. Do you have any problems in that regard?

          I’ve seen a Fujitsu D3003-s1 over here which seems an interesting alternative, 9W TDP, but still pricier and only 2 sata ports…
          Thanks for the quick replies!

          • Here’s how my sensors output looks after being up for a while; I don’t know which sensor is on which component though. I’m not doing anything at all with the video card (headless server) so I don’t expect there to be any heat issues with the GMA3150.

            # sensors
            f71869fg-isa-0290
            Adapter: ISA adapter
            in0: +1.64 V
            in1: +0.94 V (max = +0.00 V)
            in2: +1.05 V
            in3: +1.37 V
            in4: +1.03 V
            in5: +1.36 V
            in6: +1.09 V
            in7: +1.64 V
            in8: +1.59 V
            fan1: 0 RPM ALARM
            fan2: 683 RPM
            fan3: 0 RPM ALARM
            temp1: +30.0°C (high = +85.0°C, hyst = +81.0°C)
            (crit = +100.0°C, hyst = +96.0°C) sensor = transistor
            temp2: +34.0°C (high = +85.0°C, hyst = +81.0°C)
            (crit = +100.0°C, hyst = +96.0°C) sensor = disabled
            temp3: +36.0°C (high = +70.0°C, hyst = +68.0°C)
            (crit = +85.0°C, hyst = +83.0°C) sensor = disabled

            coretemp-isa-0000
            Adapter: ISA adapter
            Core 0: +14.0°C (high = +80.0°C, crit = +100.0°C)
            Core 1: +14.0°C (high = +80.0°C, crit = +100.0°C)

  10. Impressively low! Thanks for posting this.

    I’d be also using it with mythtv, so I’ll go dig a bit deeper into this gma3150 issue.

  11. Pingback: Jetway JNC9C-550-LF resume after power fail | Eric's Blog

  12. Pingback: Spinning down a WD20EARS | Eric's Blog

  13. Pingback: Watts on? | Eric's Blog

  14. I’ve been building a home nas/media server using spare parts I had laying around. My motherboard is a Gigabyte P965-DQ6, 4GB ram, 3x2TB WD Red Drives, and one 500gb drive as the boot/swap drive. The lowest power cpu I could find for the board was a single core Celeron-L 440 @ 35watts. Running the system headless, the system is pulling 120w at idle. I’m going to try a few things, but my guess is that the Intel chipset on the MB is not power friendly. The only saving grace is that the MB has a ton of sata ports. How much of a difference will a new power supply make. I have a big 450 watt supply in the machine, but would investing in a smaller wattage supply actually decrease my consumption? I always though that the PS would only supply the wattage needed. Thanks for the article! – Sean

    • A smaller, 80Plus power supply could make a difference, depending on what your current PS is rated. Yes, a power supply will only supply what’s needed, but with varying efficiency. I’ve used Antec Earthwatts 380(?) Watt power supplies with good results, it does drop the draw at the wall by a bit – and found it for around $30 on sale IIRC. The 80Plus website has a listing of all certified power supplies and their efficiency curves; you could look for one that does a good job down around 120W. A 200W 80Plus power supply might be the ticket, if 200W handles all the spinup load etc.

      Do you have the Red drives idling when they’re not in use? That might save 10W or so. Disabling unused peripherals (audio, unused sata controllers, etc) might help, but not sure about that.

  15. Most drives are rated in terms of number of spin-ups, so if you’re spinning those things up and down all the time, you’ll be saving pennies on power but costing yourself hundreds of dollars in early replacements. Funny how people think they’re saving the planet when they can see the reductions in energy directly, but think nothing of the energy costs of buying products and shipping them to their door.

    • Mr. Trollvolta –

      Thank you! I hadn’t even considered embodied energy or drive life! I thought drives had an immaculate conception! That is so funny! If I had been more thoughtful, I might have tracked drive spinup behavior over time to see if it was any worse than a typical desktop drive. Oh well! In the future I’ll be sure to run my desktop-rated drives 24/7 to extend their useful life!

  16. I think this is more a case of all things in moderation. With a NAS being used by a lot of people you may not want to have the drive spin down between each access. In a home system spinning down when their has been no access for a while is probably for the best in terms of drive life and over all usage. Heck you may even want to have the server turn off between say 2 am and 6 am if no jobs are set to run. An low power micro or even an Ardino running off a battery charged by the pc and driving a relay would do the job.

    • Yes, you’re right. A server w/ constant access should stay spinning, or at least have a long enough timeout that it doesn’t spin up & down all day.

      My server runs 24/7 because it’s performing tasks throughout the day – if it were just i.e. a music server, setting it to shut down overnight & wake up in the morning would make good sense. Simple to do w/ ACPI.

      • ACPI would only allow a sleep. You would still have a phantom load. But like I said it all depends on your usage.

        • I’m not certain; it’s a long time since I used it (had it working on an old mytobox once), but I think you can do a proper poweroff (not hibernate or sleep) and the box will wake up (i.e. do a full boot-up) at a set time. Of course there must be some degree of phantom load, but I think it’s better than a sleep. I could be wrong . . .

  17. Just to correct a minor error in this good article:

    “150W, that’s 1.3 MWh”
    should be
    “150W/h, that’s 1.3 MW/year”

    “At 18W, that’s 158 kWh/year”
    should be
    “At 18W/h, that’s 158 kW/year”

    • No… “Watts per hour” makes no sense.

      150 watts (instantaneous power) average draw, over a period of time (hours), is watt-hours, a measure of energy.

      In a year the energy consumption is 150 Watts * 365 days/year * 24 hours/day = 1314000 Watt-hours, or 1.3 Megawatt-hours.

  18. [quote]No… “Watts per hour” makes no sense.
    150 watts (instantaneous power) average draw, over a period of time (hours), is watt-hours, a measure of energy.
    [/quote]
    Yes, you COULD also write it as istantaneous power but 150W istantaneous also means 150W/h.
    Being a comparation of energy sound good to put energy on both sides.

    So I confirm that the sentence
    “150W, that’s 1.3 MWh”
    sound better as
    “150W/h, that’s 1.3 MW/year”
    especially because 1.3MWh means you consume 1.3MW per hour!!! Even writing 1.3MWh/year don’t sound very ok because you already transform power to energy saying MW/year. No need to add 2 mesure of time, otherwise would be also ok saying 1.3MWh/year/century/millennium :-)

    • If you’re consuming energy at a rate of 1kWh/h, then you are using an average of 1kWh of power over that time. That’s what 1kWh/h means, right?

      If I say 1.3MWh/year, that’s 13000000Wh/8760h, or – ~150W constant power draw.

  19. Premise1: you’re technically right using kWh/t because it’s quite common.

    Premise 2: but, the kilowatt-hour is not a standard unit in any formal system, it’s only commonly used. In the international system (SI) only W (or/kW) exists, not Wh (or kWh). This is also why I was saying 150W/h sounds better than 150wh (and mathematically are the same because h is 1 so * or / both give the same result). At best, to be barely approved by the SI we should write it as an expression: kW*h, not a unit of measure.

    Premise 3: sorry for bothering you with this techno-philosophy admittedly OT.

    What I meant is
    kWh means kW*h (where h is always 1 being the unit)
    kWh/h = kW*h/h = kW*1/h = kW*1h/(year*8760h) = kW*1h/(1*8760h) = kW*1h/8760h = kW/8760h = kW/year and this is SI compatible while kWh/year is not, despite widely used.

    from the other side
    150Wh = 150W*h
    150Wh in 1 year = 150W*h*year = 150W*1*year = 150W*year = 1.3MW/year
    commonly but weirdly referred to as 1.3MWh/year just to make more clear it is energy, not power (very useless). But at that point, if one argue that we could also infringe the SI, the same way kWh/year does, using a simpler 150W*Y. :-)
    The thing is: if we want to be fastidious on the word “energy”, the only main measure for energy approved on this planet by the SI is the Joule. The second is eV. So we should write 1.3*(3.6 * 10^9) J

    I promise to stop here without talking about the energy expressed in eV! :-)
    Thank you for your patience.

    • Sure, energy units are a mess – Joules, BTUs, kWh, therms… But I think we’ll have to agree to disagree on the use of kW/h.

      I’ll leave you with one more page that has, I think, useful analogies to water & speed.

      Watts (power) is something like speed on a speedometer (km/hr) whereas watt-hours are analogous to distance traveled (km); the integral of watts-over-time is energy (kWh), just as the integral of speed-over-time is distance traveled.

      In that analogy, talking about kw/h makes as much sense as talking about kilometers per hour per hour.

      It’s also possible that we’re just talking past each other, and we’re both perfectly correct, just misunderstanding the other correct argument. :)

  20. The most important value is absolutely clear: money! :-)
    It’s not even important if you call it bucks/quid/smackers/leftover lolly or whatever! :-))))
    See ya!

  21. Pingback: How to build a 10W, 6.5 terabyte, commodity x86 Linux server | Eric's Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.