Eric's Blog Eric's Blog
Search
How to build an 18W, 4 terabyte, commodity x86 Linux server

Can you build a Linux server for web, email, printing, and 4 terabytes of media serving purposes from commodity x86 parts, and come in under 20W? Absolutely!

If you run a server at home 24/7, and it uses, say, 150W, that’s 1.3 MWh (megawatt-hours) of electricity per year, and probably $130 or so of utility cost.  You can do better!

I recently assembled a server with the following specs:

(note, some Amazon product links follow):

Out of the box, the Kill A Watt [amzn] showed about 20W.  After running this script to further push down some power consumption, it came in at 18W:

#!/bin/bash

# Idle media disks
# (should set shortish spindown time too...)
hdparm -y /dev/sdb
hdparm -y /dev/sdc

# 100baseT saves power over GigE
ethtool -s eth0 autoneg off speed 100
ethtool -s eth1 autoneg off speed 100

# kill Wake On Lan
ethtool -s eth0 wol d
ethtool -s eth1 wol d

# set ALPM on SATA links
for FILE in /sys/class/scsi_host/host?/link_power_management_policy; do
    echo min_power > $FILE
done

# Set PCI Express power management
echo powersave > /sys/module/pcie_aspm/parameters/policy

# Try to keep 1 core idle when possible
echo 2 > /sys/devices/system/cpu/sched_smt_power_savings
# Disable the nmi watchdog
echo 0 > /proc/sys/kernel/nmi_watchdog

Ok, idling the 4T of storage might be cheating a little, but in reality, they hold music and movies, and simply do not need to be spinning all the time.  (Without these drives in the system, it comes in at about 16W.)

At 18W, that’s 158 kWh/year, or about $16.  An 88% reduction!  And, the utility savings would almost pay for the motherboard!

I don’t know if non-mobile Atoms would fare as well; my original test of an Atom motherboard was lackluster in part due to lack of power management, but then I didn’t have the efficient power supply either, so I’m uncertain.  I’d imagine that the new AMD E-350 (Brazos) motherboards would do very well too, but I’ve not been able to test one out.

And, one more bit of full disclosure – sandeen.net isn’t running on this box yet, the migration isn’t complete.  Still, quick tests of WordPress running on the platform don’t seem any slower than the current old AMD box (a 40W hog ;) ).  And if LKML keeps spamassassin really busy, maybe it’ll inch up in power a bit.  I’ll let you know when you’re reading this blog on the new machine, and what the average wattage draw is over a week or so.

So, for you geeks out there with home servers, how much power does it draw?  Would you consider something like this for your next build?

arrow38 Responses

  1. 23 mos, 3 wks ago

    I’ve had something similar in place for a NAS, sitting under the TV. No fans, small footprint, and low power consumption out of the box: http://www.tranquilpc.co.uk/T7.htm

  2. nayfield
    23 mos, 3 wks ago

    I wonder if my UPS built-in load meter reads out low (I’ll have to put it on the kill a watt to check). But I have a reading around 40w with two servers running – one is a dual-core conroe 6600 (2.4 ghz) and the other is a new sandy bridge i5-2500k. Between them they have 7 spinning drives of 1.5 or 2T each and the sandy has two 40gb ssd’s. There’s 6gb of ram in the conroe and 8 gb in the sandy.

    And my ups shows load at 75w with the 24″ panel turned on, and it bounces around 38-42w with the panel turned off. Of course neither box is doing anything right now and both systems are speedstepped down to 1.6 GHz. The power supplies are premium but nothing too green.

  3. 23 mos, 3 wks ago

    That does sound awfully low… But see what the kill-a-watt says!

  4. jone
    23 mos, 3 wks ago

    I’ve been running a via epia based server for a few years, comes in at around 50 watts with the media drives spun down.

    I’d recently been looking around for a mobile atom board with 4 sata ports, but had somehow missed this one. Thanks for the pointer!

  5. mike
    23 mos, 3 wks ago

    What do you do for redundancy? I have to admit I’m running server class hardware (Dell Poweredge 2400) but only for the hardware RAID. I have recently replaced a larger number of smaller disks with lesser numbers of large disks just to cut power usage (and I cut by almost 50% according to kill-a-watt) but I’ve lost data before and have found redundancy to be more effective in that regard to backups. My luck with backups is horrible. If I loose my wife’s email and file archive again…

  6. 23 mos, 3 wks ago

    I will actually have two 2.5″ drives in a mirror for the OS and mail/web data. The 2x 2T drives are for media, and they used to be in a mirror, but I think I will split that and rsync nightly to keep them in sync. For music & movies, a short downtime from one disk going bad is better than a filesystem problem eating both at once, I think.

  7. Anonymous
    23 mos, 3 wks ago

    Rather than the Jetway which wants ATX power and needs the Pico-PSU, might I suggest Intel’s D945GSEJT, which takes 12V DC directly? I suspect it also requires less power.

    It has only 2 SATA connectors, so you’d have to put the root filesystem on one of the 2TB drives, but still.

  8. 23 mos, 3 wks ago

    I’m surprised there aren’t more of these boards with direct 12VDC inputs, it seems quite logical at this point. The PicoPSUs are still odd beasts, but even “small” ATX powersupplies are grossly oversized at this point…

    But yes, the limited SATA connectors would be an issue for my usecase, I really wanted 4. I suppose you could always add a PCIE sata card too, but I wonder how many drives the motherboard can supply power to?

  9. Amit
    23 mos, 3 wks ago

    Disclaimer: I work with several ARM vendors as part of linaro.org

    I always feel a bit frustrated when I read articles about low power servers @ 20W and there is no mention of ARM-based devices. It reminds me of our failing as part of the ARM community.

    Here are a few random links from googling. [1] talks about the more recent boards (still under 5W) and [2] about a seriously underpowered one by today’s standards that still seem to be enough for some people.

    One thing I’ll readily admit is disk performance or lack there of with these boards since they only take USB hard drives. If that isn’t a constraint, then you should definitely look at these.

    [1] http://jeffbastian.blogspot.com/2011/04/my-pandaboard-has-arrived-it-was-only.html
    [2] http://nslu2watt.dyndns.org/

  10. 23 mos, 3 wks ago

    Hi Amit – I know ARM would be lower power, and I would love to run on ARM. Really! I have a sheevaplug, and it’s fantastic at ~6W, but it can only do so much without more connectivity.

    You’re right – I can’t add storage to the ARM solutions I see, other than via external USB disks, and the accompanying performance issues, tangles of cables and less-efficient wall-warts.

    Why are there no mini-ITX formfactor ARM boards with 4 SATA connectors? Get one of those out there for under $200, and you’d have a winner, I think.

  11. Amit
    23 mos, 3 wks ago

    > Why are there no mini-ITX formfactor ARM boards with 4 >SATA connectors?

    4 SATA connectors! You do know how to inflict pain. :)

    We finally have one board (imx53 quick start) that has one SATA connector. Find details here: http://www.linaro.org/low-cost-development-boards

  12. 23 mos, 3 wks ago

    > 4 SATA connectors! You do know how to inflict pain. :)

    Dumb, but honest question: Why is that so painful? By the time you have one, isn’t at least the second “almost free?” :)

    nb: that board is not available now, and when it is, it’ll be $394. (Edit: oops, bare board is $199; but, still sold out) Sorry, that just doesn’t work for my server needs … But I want to believe! I really do hope ARM becomes useful in this space.

  13. Amit
    23 mos, 3 wks ago

    We’ve been asking for SATA for years, hence the pain. And after so many years we’ve finally managed to get 1 SATA connector.

    But ARM is starting to look at the server space now, so things should change rapidly. (Fingers crossed)

  14. 23 mos, 3 wks ago

    When you get a 2- or 4- SATA board, send me one and I’ll blog about it…*grin*

  15. nayfield
    23 mos, 3 wks ago

    Turns out the UPS is massively off. The sandy bridge idles around 65-68 watts with 16gb ram.

  16. Navas
    23 mos, 3 wks ago

    You can even bring it to 16W with the N455 version (single core atom)

    http://www.newegg.com/Product/Product.aspx?Item=N82E16813153189R

    I hope for a file server, we don’t need multi cores

  17. Jay
    20 mos, 3 wks ago

    What did you use for a case? I’m having a hard time finding a mini-itx case that holds more than one drive.

  18. 20 mos, 3 wks ago

    It’s a LIAN LI PC-Q11B – I’ve updated the parts list, sorry about that!

  19. 17 mos, 4 wks ago

    18Watt? Wow, great job Eric ;)

  20. PJBrs
    17 mos, 1 wk ago

    Great article!

    Are you using the case fan with this board? And does it ever completely turn off with the bios smart fan configuration? I’m thinking of buying this board, but I wouldn’t want to hear any fan noise if the system is completely idle, while at the same time it seems that a case fan is required when the system is under high load.

  21. 17 mos, 1 wk ago

    The Lian Li case I got did come with a large fan on the front. To be honest I have not paid attention to the fan; I’ll try to look at the bios settings soon and reply again with more info. Fiddling with the sensors infrastructure might also be useful there.

    The board is a bit pricey still; I’d be interested to know how an AMD Brazos board would fare power-wise; they are available at decent prices now.

  22. 17 mos, 1 wk ago

    FWIW f71882fg.ko (F71882FG Hardware Monitoring Driver) handles the fan sensors on the board; as it sits idle right now the main fan is at 675RPM. I think the smart fan control is on in the BIOS.

  23. PJBrs
    17 mos, 1 wk ago

    And at what temperature does it idle? I wonder whether it would be possible to configure fan control so that the fan is off when idling.

    By the way, I read some problems about the GMA3150 running hotter on linux than on windows. Do you have any problems in that regard?

    I’ve seen a Fujitsu D3003-s1 over here which seems an interesting alternative, 9W TDP, but still pricier and only 2 sata ports…
    Thanks for the quick replies!

  24. 17 mos, 1 wk ago

    Here’s how my sensors output looks after being up for a while; I don’t know which sensor is on which component though. I’m not doing anything at all with the video card (headless server) so I don’t expect there to be any heat issues with the GMA3150.

    # sensors
    f71869fg-isa-0290
    Adapter: ISA adapter
    in0: +1.64 V
    in1: +0.94 V (max = +0.00 V)
    in2: +1.05 V
    in3: +1.37 V
    in4: +1.03 V
    in5: +1.36 V
    in6: +1.09 V
    in7: +1.64 V
    in8: +1.59 V
    fan1: 0 RPM ALARM
    fan2: 683 RPM
    fan3: 0 RPM ALARM
    temp1: +30.0°C (high = +85.0°C, hyst = +81.0°C)
    (crit = +100.0°C, hyst = +96.0°C) sensor = transistor
    temp2: +34.0°C (high = +85.0°C, hyst = +81.0°C)
    (crit = +100.0°C, hyst = +96.0°C) sensor = disabled
    temp3: +36.0°C (high = +70.0°C, hyst = +68.0°C)
    (crit = +85.0°C, hyst = +83.0°C) sensor = disabled

    coretemp-isa-0000
    Adapter: ISA adapter
    Core 0: +14.0°C (high = +80.0°C, crit = +100.0°C)
    Core 1: +14.0°C (high = +80.0°C, crit = +100.0°C)

  25. PJBrs
    17 mos, 1 wk ago

    Impressively low! Thanks for posting this.

    I’d be also using it with mythtv, so I’ll go dig a bit deeper into this gma3150 issue.

  26. LWATCDR
    13 mos ago

    Well here is an ARM board with two SATA II slots and Giga-e. http://www.embeddedarm.com/products/board-detail.php?tab=options&product=TS-7800
    The problem is that it is a little expensive, it has limited RAM, and has a lot of IO that I NAS just does not need.
    It would work but no cigar yet.

  27. 13 mos ago

    Thanks! I actually have one of those boards from an old embedded appliance, but I haven’t really played with it…

    -Eric

  28. LWATCDR
    13 mos ago

    BTW that board does offer an 802.11n option but if you added a USB to ethernet dongle so you have a second serial port and their wifi adaptor or a USB wifi adaptor you could make this into a SAN, Firewall, Router, WiFi router as we.
    With all the serial ports that it offers you could also add a Zig Bee radio for some remote data collection projects.

  29. Adam Baxter
    11 mos ago

    The problem I have with the ARM SoC based “small PCs” is that a lot of the time the SATA ports are connected internally to the USB… at least that’s the case with the Dreamplug. Other than that I’m very happy with my Dreamplug.

  30. Antonio C.
    9 mos, 3 wks ago

    What about a QNAP TS-412 ?
    http://www.qnap.com
    http://www.qnap.com/en/index.php?lang=en&sn=822&c=351&sc=514&t=522&n=3415

    I am trying to find a good low consumption motherboard, but I always end up choosing the QNAP products for home/office. It has everything you need, it’s Linux based and I am impressed with all the features it has.
    The cost of a mini-itx, plus a processor, plus a power adapter, plus a case remains high.

  31. 3 mos ago

    I’ve been building a home nas/media server using spare parts I had laying around. My motherboard is a Gigabyte P965-DQ6, 4GB ram, 3x2TB WD Red Drives, and one 500gb drive as the boot/swap drive. The lowest power cpu I could find for the board was a single core Celeron-L 440 @ 35watts. Running the system headless, the system is pulling 120w at idle. I’m going to try a few things, but my guess is that the Intel chipset on the MB is not power friendly. The only saving grace is that the MB has a ton of sata ports. How much of a difference will a new power supply make. I have a big 450 watt supply in the machine, but would investing in a smaller wattage supply actually decrease my consumption? I always though that the PS would only supply the wattage needed. Thanks for the article! – Sean

  32. 3 mos ago

    A smaller, 80Plus power supply could make a difference, depending on what your current PS is rated. Yes, a power supply will only supply what’s needed, but with varying efficiency. I’ve used Antec Earthwatts 380(?) Watt power supplies with good results, it does drop the draw at the wall by a bit – and found it for around $30 on sale IIRC. The 80Plus website has a listing of all certified power supplies and their efficiency curves; you could look for one that does a good job down around 120W. A 200W 80Plus power supply might be the ticket, if 200W handles all the spinup load etc.

    Do you have the Red drives idling when they’re not in use? That might save 10W or so. Disabling unused peripherals (audio, unused sata controllers, etc) might help, but not sure about that.

  33. John Trollvolta
    3 wks, 3 days ago

    Most drives are rated in terms of number of spin-ups, so if you’re spinning those things up and down all the time, you’ll be saving pennies on power but costing yourself hundreds of dollars in early replacements. Funny how people think they’re saving the planet when they can see the reductions in energy directly, but think nothing of the energy costs of buying products and shipping them to their door.

  34. 3 wks, 3 days ago

    Mr. Trollvolta -

    Thank you! I hadn’t even considered embodied energy or drive life! I thought drives had an immaculate conception! That is so funny! If I had been more thoughtful, I might have tracked drive spinup behavior over time to see if it was any worse than a typical desktop drive. Oh well! In the future I’ll be sure to run my desktop-rated drives 24/7 to extend their useful life!

  35. Lwatcdr
    3 wks, 2 days ago

    I think this is more a case of all things in moderation. With a NAS being used by a lot of people you may not want to have the drive spin down between each access. In a home system spinning down when their has been no access for a while is probably for the best in terms of drive life and over all usage. Heck you may even want to have the server turn off between say 2 am and 6 am if no jobs are set to run. An low power micro or even an Ardino running off a battery charged by the pc and driving a relay would do the job.

  36. 3 wks, 2 days ago

    Yes, you’re right. A server w/ constant access should stay spinning, or at least have a long enough timeout that it doesn’t spin up & down all day.

    My server runs 24/7 because it’s performing tasks throughout the day – if it were just i.e. a music server, setting it to shut down overnight & wake up in the morning would make good sense. Simple to do w/ ACPI.

  37. Lwatcdr
    3 wks, 2 days ago

    ACPI would only allow a sleep. You would still have a phantom load. But like I said it all depends on your usage.

  38. 3 wks, 2 days ago

    I’m not certain; it’s a long time since I used it (had it working on an old mytobox once), but I think you can do a proper poweroff (not hibernate or sleep) and the box will wake up (i.e. do a full boot-up) at a set time. Of course there must be some degree of phantom load, but I think it’s better than a sleep. I could be wrong . . .

Leave A Comment

Notify me of followup comments via e-mail. You can also subscribe without commenting.