Designing the home NAS

In an earlier installment, I pointed out that popular branded solutions are surprisingly expensive for low-performing hardware.  Reviews indicate that they have rather poor performance.  So for my comparison, I’ll use the Synology DiskStation DS1513+, which reportedly has good performance, more than what a single-link gigabit Ethernet connection can handle.

It has quite a few things in common with the home-made solution:  Multiple Ethernet ports that can be used for redundancy or increased throughput, the ability to host related servers as “apps”, and not user-friendly enough for novices to do the initial setup.

While I was doing this, the Synology DiskStation could be found for $830.  It contains a dual-core Atom D2700 running at 2.13 GHz and 2GB of DDR3 RAM.

Now, there are two ways to approach this.  Clearly a competent file server can run on low-end x86_64 processors with a small (by today’s desktop standards) amount of RAM.  The original FreeNAS was commonly used with hand-me-down hardware.

But, times have changed.  The new FreeNAS, a rewrite by iXsystems, was designed with more modern concerns in mind:  RAM is much cheaper now and the system can be more capable and easier to write if it doesn’t have to cope with low-RAM installations.  In addition, the safety of ZFS against mysterious data corruption relies on the RAM not having mysterious corruption too, and should be used with ECC RAM.  Then comes dire warnings about Windows file shares (CIFS) being single threaded and thus needing a fast CPU (as opposed to multiple slower cores), and features such as encryption demanding ever more CPU performance.  Oh, and the Realtek NIC used on many consumer motherboards is not good for FreeNAS; it needs an Intel NIC.

In short, I’m looking at a server-grade system, not a typical desktop or “gamer” enthusiast system.  What you don’t need is fancy overclock support, sound, lots of slots and multi-video-card support, etc. so a low-end server board is actually about the same price as a “fancy” desktop motherboard.

In particular, the Supermicro brand comes highly recommended.  I could have gotten an X9-series server motherboard and put a Xeon E3 v2 CPU on it.  But why stop there?  I spent more to go with the newer X10-series board and a Xeon E3 v3 “Haswell” CPU.  The X10-SL7-f in fact contains an 8-channel SAS controller as well as the usual 6 SATA channels, sprouting a whopping 14 SATA connectors on the motherboard.  It also features IPMI 2.0 on its own dedicated network port, which is a wonderful feature and I’ll have more to say about it later.

So without further ado, here is the breakdown of my build:

Parts List

Item Description
Price
ICY DOCK MB153SP-B 3 in 2 SATA Internal Backplane Raid Cage Module$63.99
Intel Intel Xeon E3-1245V3 Haswell 3.4GHz LGA 1150 84W Quad-Core Server Processor$289.99
SUPERMICRO MBD-X10SL7-F-O uATX Server Motherboard$239.99
SeaSonic SSR-360GP 360W ATX12V v2.31 80 PLUS GOLD Certified Active PFC Power Supply New 4th Gen CPU Certified Haswell Ready$59.99
Fractal Design Define R4 Black Pearl w/ USB 3.0 ATX Mid Tower Silent PC Computer Case$99.99
ZALMAN CNPS5X Performa 92mm FSB (Fluid Shield Bearing) Powerful Cooling Performance CPU Cooler$19.99
2 × 8GB PC3-12800 DDR3-1600MHz ECC Unbuffered CL11 HYNIX Memory178.92
Total without drives$952.86
WD Red WD40EFRX 4TB IntelliPower 64MB Cache SATA 6.0Gb/s 3.5" NAS Internal Hard Drive -Bulk3×$189.99 = $569.97
Seagate ST4000DM000 Desktop 4TB 64MB Cache2×$155.49 = $310.98
Total for Build$1833.81

The raw power seriously outclasses the DiskStation, and is only $120 more.  With the X9/v2 option, it would have actually been less.

Oort build - inside

Above is the result, Oort, with the side open.  You can see the stack of 8 drive trays, and the large heat sink over the CPU.

view-_MG_2530

Here is a front view.  The grill along the edges allow air intake from the front.   The blank front face is imposing and mysterious… I wonder if I can get some artwork over it?

view-_MG_2534

And finally, with the front panel open.  There is foam sound-dampening on all the case surfaces including the inside of this door.  The ICY-Dock hot-swap bays are now accessible.  I plan to use these for backing up and mounting off-site volumes while they are resident.  The main drives require side access, which is simply a matter of removing two thumb screws.

Now back to the details.  The X10 (rather than X9) series mainboard allows the use of the newer Haswell processors, which run cooler and save power.  The onboard SAS saves what would be a hundred dollar PCI card, and is much easier as well since it provides common SATA-compatible connectors.  And finally, this motherboard has the wonderful IPMI 2.0 with full KVM-over-LAN.

For the CPU, I looked at the the chart in Wikipedia, along with the prices and availability at NewEgg.  I chose the lowest (cheapest) Xeon E3 that had onboard graphics and hyperthreading.  Why do I need onboard graphics if the system doesn’t have a monitor?  I think that the monitor-over-LAN feature still requires an actual VGA; it doesn’t emulate one, but just captures the output.  There is a more primitive remote management feature that allows for a TTY-style console (also over LAN), but I don’t think that helps with initial BIOS screen stuff.  Also, with the standard built-in GPU I can use it for computation other than drawing graphics.  Maybe it will accelerate other software I run on the box at some point.

I’m keeping the box in a closet which besides building up heat from the machines gets afternoon sun on the outside wall.  The closet is warm in the summer.  My experience with the stock cooler that comes with the CPU is that it’s loud or even inadequate.  Looking through NewEgg, I looked for this style with low noise and a good price.  I normally like this style in part because it takes a standard square fan which can be updated and replaced, but the Zalman is known for quiet fans too.  I mounted it, not with the thermal grease that it came with, but with Phobia HeGrease, carefully applied and spread.

The RAM was not available at NewEgg.  Apparently ECC but not Buffered/Registered also is uncommon.  Buffering is used to facilitate having many more memory sticks on a board, which is not the case of this server-but-desktop board.  I found it at a specialty RAM site, www.memoryamerica.com, which has a wide selection.  To be on the safe side, I looked at the brands that Supermicro had tested on this board, and took the cheaper of the two.  16GiB uses two of the four memory slots, so it can be doubled in the future.

I use Seasonic power supplies, and that’s another story.  I looked for “Haswell support”, which enables a new improved stand-by mode.

Now for the case:  Some mentions on the FreeNAS web forum led me to Fractal Designs.  I followed up by reading reviews and the manufacturer’s web site.  There are a couple models that are so similar that I wonder what the difference is!  Since there is no direct explanation, it takes reading the specs very carefully and comparing the dimensions to spot the real differences.  This R4 features an internal stack of 8 HDD trays (with the anti-vibration mounting) plus two half-height 5¼″ external bays.  If you include two SSDs stuck elsewhere, that is 13 drives total, which is nicely close to the motherboard’s support of 14.

I chose an option with two external bays so I could fit a 3-disk hot-swap backplane.  Here I went with the name-brand ICY-Dock and a with-tray design, because I had trouble with trayless on Mercury.  So using the front-loading drive bay requires the use of two mounting screws, which is not very handy as it turns out.

Worse, the “2 half height bays” is a little exaggerated.  It’s more like 1.95 half height bays, as a large bulge protrudes into the area where the bottom bay should be.  I had to remove the bottom piece of sheet metal from the ICY-Dock in order to squeeze it in; this also got rid of the normal mounting ears.  I’ll make a bracket some day (a perfect job for a 3D printer), but it fits tightly and is not heavily used, so I left it without screws for the time being.

Other than that, assembling was easy and straightforward.  Testing proved interesting and adventuresome, and I’ll tell you about that later.

4 thoughts on “Designing the home NAS

  1. Bob Hyatt

    John,

    I have read the two articles you created on the topic of backups, for today and the near future. I especially appreciate that you found a solution rather quickly. I am ‘jealous’ of that, as I am nowhere near a resolution.

    Part of the issue for me is there are a lot of products available using only a few file systems between them. In my opinion, the most modern file system appears to be ZFS. They have dealt with concepts and IT history in what I consider a good way. The issue is to determine which variety of ZFS is worth my time.

    There has not been a comparison chart of the various file systems, yet. Yes, there are a few listings:

    http://en.wikipedia.org/wiki/List_of_file_systems A listing of various file systems, but not a comparison
    http://en.wikipedia.org/wiki/ZFS for the various flavors of ZFS
    http://open-zfs.org/wiki/Platform_code_differences identifies differences at a high level

    The good news on the last web-site is this means the ‘open software’ idea is working. Code is reviewed in many ways, that can lead to great results.

    You asked if I have a website. I do, but it was intended for a non-BobHyatt purpose. It was my experiment with Joomla, as I thought I had a few things to say. The ‘club’ has chosen another way (Yahoo) as my experimentation was taking too long. So I have an available site, with a name that could mean almost anything. I do have forums working, so I could open it up for that purpose. I have paid for 2 years of running the production site, in Singapore. I paid for 5 years for the website name from Go Daddy. I thought it rather odd that I can only ‘lease’ the name, for a limited time, not purchase forever. hmmm?

    As time goes by, I am reverting back to my normal state of introvert. In order to be successful at the worksite I had to ‘pretend’ to be an extrovert, and it cost me. So the result is I seldom go to the efforts you have read. I prefer to investigate and consider, at length. The conversations I have started was just another way to get different opinions for me to investigate.

    I asked FreeNAS to tell me what differentiates them from all the other products and either they don’t know, or they don’t want to say. Seems like a reasonable question to ask, Yes?

    In my experience, I have been able to reduce any system to its most basic list of attributes and or features. The ‘backup systems’ arena has broken up into some groups. I have been in the most basic one, in my use of Icy Dock usb stations, in which I control what is backed up and when (done manually).

    Drobo is another level, but it is very slow.

    The most expensive (in initial cost and in replacing hard drives) are the large raid units. I saw one via YouTube that had 60 drives in it. The big issue for me is how they kill the drives in their constant movement of data around. An enterprise can afford that, but I cannot.

    I need the resulting system / tool I end up with to do only the essentials, gently, yet timely. Rather vague terms, but they can be detailed. I saw a YouTube, in which two raid array systems were shown. With the same number of drives, in same U4 configuration. The only difference was the software running the file systems.

    One was constantly busy, the lights on the hard drives are busy, busy. The other, doing the exact same work, is working less hard and the hard drive lights do not light up nearly as much as the first. Search for “Openfiler SAN Cluster vs Ubuntu SAN Cluster” and you will see the difference.

    I would prefer one that is NOT constantly working the drives. In addition to the size differences in the disks I have, I also have ‘type’ differences. WD Green is the most prevalent. I would bet they would die first in the OpenFiler SAN / NAS. I can’t afford to be replacing disks every month, or even every year. I don’t have that much work for the system to do. That is why I had planned shutting it down except for when I need to do a backup.

    However, there are other benefits to such a device. I do not have such a network today, but I do a sneakernet kind of thing for the videos and music. I put the data onto USB 3 devices which are carried to my media players in my house. This acts as another backup, albeit a small subset of the files I keep. If it was a good, reliable, and gentle system I could wire it directly to the media players, and to the wireless network I have in the house, and others could watch anytime.

    However, this would be a low priority use, until after I have resolved the backup issue. One ‘good’ thing, is it appears I have double the number of drives I actually require for the data I have now. This gave me the idea of a mirror. e.g. two complete sets of backups in one device. sounds good to me…

    But that won’t happen until I decide which file system to use. That will determine which OS to use, which motherboard to buy, which cpu, which kind of memory, and so forth.

    I would prefer something running on Linux. I took a look at FreeBSD, and was annoyed almost immediately at the jargon. Too many IT folk are showing off and trying to intimidate others with their ‘brilliance’. Been that way since I began working in IT in 1970. By NOT doing that, I was able to get things done, decline multiple suggestions of moving into management, and make a good salary. And keep sensible.

    So thank you for being willing to read my words. This one is not as focused as I would have preferred, but I got the notion you wanted this kind of ‘diatribe’ to get another view of the issues.

    Reply
  2. Bob Hyatt

    John,

    You invited me to write more on this topic. I believe I will do so, but in parts, rather than in a 20 page university paper. I will start with an overview of the ‘desired’ home server backup, vs what seems to be available.

    For most homes and small businesses, the primary desires / goals are (in my view):

    A. secure backups. In this case I add, secure from failure for at least 5 years!

    B. Easy to do backups. I do NOT mean automatically scheduled backups. They automatically back up things you are NOT interested in. Or that will be useless to you if there is a problem. For example, do you really want to back up all the Operating System files, settings, etc. Things that change a lot, and will occupy a lot of space on your backup drives? I do NOT.

    C. Allow you to determine what is important to You!
    Not every photo or video is important to you. Not every PDF is valuable to you. Not every MP3 or FLAC is valuable to you. So you need a way to differentiate between valued and chaff. And for the ‘tool’ to remember. OR for you to keep the data you want to keep separately from the chaff. Then have the ‘tool’ look only at the desired data.

    D. Allow you to add storage, as needed, easily. Adding a new hard drive should not be ‘a hope and a prayer’ kind of task.

    E. If I should desire to change from one backup product or tools to another, the existing should NOT act as a barrier or gatekeeper. (as in keeping your data as prisoner).

    F. The running of the backup device or server should NOT wear out the disks, ever! The issue with the video I described in the earlier ‘letter’ here was that the one always busy was attempting to make the ‘distribution’ of data on the drives “perfect”.

    If you are a corporation or a government agency, such wearing out of the disks would be considered ‘cost of doing business’. And they have the budget for this task. A home user or a small office does NOT have a budget for this willful destruction. Further, “Perfect distribution” is NOT what a home user or small office is looking for. They are looking for a safe place to put their data, and the exact location is not important.

    Home or small business users will seldom read the backups. Perhaps, not ‘seldom’. But the reality is you will do plenty of writes to update the backup. In some cases you might do more reads (if you share the data with media players, other computers on your network, etc).

    One more difference between the products that are most sold in this arena? Large organizations are also looking for quick access to the data. E.g. The usage of this kind of thing is for ‘backup or archival’ purposes for the home and small business, whereas the large organizations are using the data put onto such a device as production data and they expect lots of access and lots of updates.

    SAN and / or NAS systems in the large organizations are for processing production data, not for backups. So you begin the issues with a fundamental difference in the use of these devices. This is partly why the components tend to be expensive.

    The good news for the home or small business is the time tested components used by the large organizations, while expensive, are likely to last longer than your need.

    If I get some feedback on this introduction, I will introduce ‘task vs tool’ in the next ‘letter’ I write.

    Reply
  3. Bob Hyatt

    John,

    I have continued my research and I have stumbled into some roadblocks. Issues that make me think of a deliberate effort. annoying. I won’t detail just yet. First, an overview of the ‘international’ conditions that exist.

    Digital Rights. or more properly (IMHO) Content Rights. When I purchase a DVD, CD, or anything else in which I have something I value, I expect to keep the ownership (via the purchase) forever. However, in practical terms this turns out to be more difficult.

    I purchased a movie on VHS tape in the early 1990s. When DVDs came out, I purchased the same movie. It was supposed to last, but after a few playings, the media was scratched, etc. So I purchased another dvd, which was supposedly ‘improved’. I have converted the DVD into a media file, and put the DVD away, so it will not be damaged.

    I put this movie on my pc’s hard drive, during the conversion process. After converting some hundreds of dvds, I have a collection that I like. I copied the movie media files onto a USB drive, which I plug into a media player (making another digital copy).

    Now, we come to the problematic part. While ‘fair use’ allows me the rights to do all the things I mentioned above, new attempts at limiting such rights keep appearing. Almost every year, another attempt it made to make what I have done illegal. The hint is that someday they will sell movies in a media form and they do NOT want you to already have such files. Nor do they want you to share such files.

    E.g., right now I can share the dvd’s I have, and they can’t do anything about it. But sharing media files? grey area?

    Ok, so what does this have to do with making a secure backup server, from which I might connect the server to my media player? Technical obstacles.

    You want to purchase a ready made backup / server combination? Expensive, and minimally powered. And the parts are ‘limited’ in that as standards change, they do NOT change. Example: The switch from 2 terabyte hard drives to 3 terabyte hard drives has been difficult. Such devices do not have the capability to ‘grow’ as standards change.

    Next, the cpu hardware in such devices are ‘just enough’ to do the job, at the time you purchase them. There was a time when a movie, due to poor conversion capability were very small. And you had to have the correct codec (coder / decoder) software to read the file and then to convert it into the signals that your tv could understand. Then the media files became very large, and again, you had to have the right codec’s in order to play those files. Now, new codecs have made the files much smaller, yet are high definition, and again you need to have the correct media player codecs. These devices don’t offer an easy way to do that.

    Next, the sizes of these file system devices are small. Having only two or three hard drives in them may seem like a lot, but in reality, once you get into media file ‘collection’, you can easily outgrow these. And one purpose of a server that can also act as a backup, is you can put files on it that don’t exist on your normal, day to day, pc. The point here is you enjoyed a specific movie on your pc, but once you put it onto the backup, you removed it from your main, day to day, pc.

    If you use packaged backup software, once you deleted the media file from your pc, it likely also deleted it from the backup. Is that what you wanted? If you managed to keep the file, how long before you fill up the device’s two or three drives?

    So, what options remain?

    Latest ‘fad’ has been ‘cloud storage’. Saving data on the internet is not new, but it has become a bit easier with the ‘cloud’ interpretation. There are issues. Who owns the data once it has been uploaded? There was quite a stink a few years ago, when a popular photo storage site revealed that once the pics were uploaded, that company owned all the rights to the picture files. ooops…

    If Digital Rights become more restrictive, do you dare backup a movie media file on the cloud? Or a cd that you converted into mp3 files? If the cloud storage company receives a demand from some legal entity demanding that all files named ‘I-Love-This-Music.mp3′ are to be removed, what will happen? Whose ‘rights’ will take precedence? Or will there be a time limitation on any media file? Or can the media file’s usage be revoked? Do you remember the time Amazon, who had sold pdfs of a popular book, using their digital reader hardware, deleted all copies of that one file on the many thousands of people who had the file? So they sold the readers, then sold the book’s pdf, then deleted the file. Is this a business plan?

    The end result is you truly do need to have a private, secure, and accessible place to put your backups of files, whatever kind and sort those files are. If you agree with this assessment, what do you do now?

    The most obvious is to build or buy another pc, putting an operating system designed for server functions. Microsoft has one. But is it secure?

    I recently had an issue with my ISP (Internet Service Provider). I had just onboarded with them and was having an issue getting and staying connected. When I called their help desk, they asked that I turn on my PC. I said it was already on. They said, they could not see my pc. (prior to this part of the conversation I had established a connection, and gave them the ip address). They then said, I must turn on a Windows OS PC. I have one of those, but my main pc has Ubuntu Linux. Hmmm?’

    I turned on the Windows pc and connected, and I heard an audible ‘ahhh’ from them. They then read off information about that pc in their efforts to confirm it was my pc. In spite of having firewall and anti-virus software they easily reached into my Windows PC. They could NOT do that on my Ubuntu Linux pc, because that OS considers such ‘activity’ dangerous and rude.

    So, the lesson here is that the Windows home server software is likely to NOT be secure. So, exactly when will they allow active deletion of media files? A directive from some legal source (lawyers for music companies, or the U S Justice department?) could tell Microsoft to remove every copy of a specific file on any server having their software, upon the next connection to the internet for a ‘windows update’. Scary.

    So exactly what can you do? There are other OS’s for servers. But for most persons managing Windows pcs are tough, and working with some other OS is frightening. It does not help that most of those other OS’s came from extreme ‘geeks’ who don’t know how to talk ‘normal’. Yes, there are good reasons why the hard drives have such awful names / codes, but it does not help the normal person who just wants to make backups.

    As I began this adventure, I wandered into ‘continuous debate’ land. e.g. people doing mini-flame-wars in their efforts to ‘make a point’. Sometimes the discussion is informative and in one case, definitely useful. This particular discussion had to do with ‘memory errors’. The issue is that while ZFS is a great file system, containing many efforts to ensure good copies exist, it can be a victim of a little publicized thing called ‘memory errors’.

    Memory errors are well known in the computer industry. I remember learning about it in class, almost 40 years ago. Every mainframe has a kind of memory called ‘ECC’ [Error Correcting Code], which means it has a way to verify that what is in memory is good and has no errors. Of course, there are two types of this ‘monster’, also. One can detect the error, and the other can detect the error and fix it automatically. This sounds great, but what can cause an error in memory?

    As electronic circuits have become smaller, they gradually became susceptible to stray atomic particles that can change the data. Most familiar is the gamma particle. When ever a gamma particle passes through a specific memory cell, it can change the charge in that cell, which changes its intended meaning. ECC will detect this change and fix it.

    Sounds good, eh? Almost NO personal computer has ECC memory! In fact, when I tried to do a search using ECC as the search item, I was unable to find motherboards that ONLY had ECC memory. e.g. the searches I did found non-ECC and ECC equally. If you go to a company that makes motherboards, they have selection criteria to help you determine which motherboard you want to know about. ECC is NOT one of those criteria. This means you have to look at every motherboard’s specifications to see which support ECC functions. There are some ‘general’ categories that can help. Assume that no ordinary motherboard (like an enthusiast motherboard) as ECC support. If the motherboard is in the WS [Work Station] category, it might support ECC. if the motherboard is in the server category, it is likely to support ECC.

    I will stop for now… If you are interested, let me know and I will keep on typing my research.

    Reply
  4. Bob Hyatt

    John,

    I am happy to tell you that the market has provided exactly the system we need for a great Home NAS.

    The obstacles for the home builder were something like this:

    Motherboard – ECC Memory?
    Raid Card, in which you would NOT use the raid function, but use the multiple connections to the hard drives.
    CPU
    CPU Cooler

    Taken together you needed a single motherboard that would support a whole bunch of drives, ECC memory, supporting a powerful CPU, and yet not require massive cooling.

    The ASRock C2750D4I 201404 motherboard can do all the things required for a home (or service center) NAS.

    The price for this mother board is over USD 400, when I last checked. Seems too high? But with this board, comes the CPU and the CPU Cooler. I also comes with 12 SATA Ports, and built-in Video. This combination of parts, on MB means savings.

    This particular motherboard has a Octa-core CPU. There is also one that is a Dual Core CPU, but I suggest this one. Why? For ‘normal’ use a Dual Core CPU would be ‘enough’. The time you would max out the CPU is when you have to ‘rebuild’ one or more drives. Both the drives and the CPU will be heavily used. The quicker you can do this work (meaning the CPU usage) the quicker it will be done, with less chance of a new problem.

    If you search YouTube you will find some reviews. I consider it an answer to the many issues I outlined earlier, in regards to the actual build of the NAS.

    Best regards, Bob Hyatt

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>