Help with implementing a hard drive

Discussion in 'Digital Sources' started by L4T, Jan 7, 2019.

  1. L4T

    L4T New Member

    Messages:
    3
    Greetings all,
    I have recently converted all my music and have it loaded on an external hard drive (15k-20k songs).
    Is there a way to play music on the TV/home stereo (WiFi on TV and Bose stereo with Bose Soundtouch), cell phones (even when cell phones are away from home), and each computer in the house?

    I am trying to understand iTunes/Amazon/Google etc. but not sure if any of this allows for music on multiple devices. Also, if there is a platform, is it only .mp3 or can it be lossless formats?

    Basically I am wanting a way for the hard drive to be an old cd changer that holds hundreds of discs, able to select songs and albums....maybe even shuffle songs and play them on any device/main stereo in the house as well as have that music on the go with the cell phones.

    Thank you for your time and guidance.
     

     

    Please register to disable this ad.

  2. cpt_paranoia

    cpt_paranoia Super Member

    Messages:
    1,061
    Check the capabilities of your router; it may have a USB port that will take a HDD, and create a file server and DLNA media server (a "poor man's NAS"). You would then access either the file server with a media management/player tool, or the media server using a DLNA media player tool.

    Playing when away from home is a little bit harder, although some routers even provide remote access.

    I have my music on a WD MyCloud NAS drive, connected to my router. All my devices at home can access my media on this NAS, either accessing the file server or the DLNA media server that the NAS provides. I can access the file server remotely, and I could install a copy of Bubble Server on it to provide a remote proxy server to allow me to access the DLNA server remotely, but at the moment I don't do that.

    http://audiokarma.org/forums/index....your-hi-fi-system.826709/page-5#post-11814431

    This may be a more useful sub-forum:

    http://audiokarma.org/forums/index.php?forums/pcs-music-servers.122/
     
    Last edited: Jan 7, 2019
  3. L4T

    L4T New Member

    Messages:
    3
    Wow. Thank you very much.
    I will read....learn....and, implement.
     
  4. RTally

    RTally Speaker addict Subscriber

    Messages:
    857
    Location:
    Knoxville, Tennessee
    Make sure you have a backup of all the music on that drive.
     
  5. sgmlaw

    sgmlaw Well-Known Member

    Messages:
    651
    You are raising a lot of different issues. Some as to file format and container, others as to storage protocol, networking and best practices. And yet others as to how to serve the data to your clients. But don't fret, you are not the first one confronting this. I have been running my own digital library off of HDD for over 15 years now. In 2001, I started ripping my entire library of CDs to HDD, and have never looked back. I now have over 70,000 audio tracks on HDD. All of it is accessible from every computer, every mobile device, every TV and every stereo. And a number of portable wireless players as well. I think that is what you are talking about.

    First off, the data. If you have not already, you are going to be putting a lot of time into creating, tagging, organizing and managing your media data. Perhaps thousands of hours. One head crash on a single HDD, and it could all disappear. No matter which form you have ripped or converted your files into, they should be kept in some fault-tolerant fashion, with adequate back up in case a mistake happens. A single HDD is asking for disaster. Ideally, your working data should be on a RAID array of some form. While RAID10 is best, many people can get by with RAID5 even today. Whether you elect to set up a software or hardware array is up to you. I keep our working digital media (what is accessed and managed) in a HW RAID enclosure connected to our server machine by Thunderbolt link. You may not need to go to that length, but you absolutely need to keep your data in a fault-tolerant storage medium.

    As far as backup, you can keep a local backup, an off-site (cloud) backup, or both. I do both, with a networked NAS for backup of the working data, and a cloud service off the server machine. In the event of a local loss, I can recover locally in a few hours. And if the house burns down, I can get replacement drives delivered to me overnight if I need them. Think about a backup in the event you accidentally erase or corrupt a media file on your working drive. It happens all the time.

    Next is how the data is going to be physically accessed by your clients. With multiple client types, the best way is through a home network. Because your data needs to be served to these clients, you will need to tether the data to a networked machine that is capable of running one of the primary server platforms or services in use today, or be network-accessible to a machine that can run such server applications and services. That is either a NAS or a machine that you would designate as your server machine. Many NASes are capable of running many of the major music server applications. Most all freestanding home computer systems are as well. The data does not need to be on the machine directly, but it needs to at least be accessible to that machine. I tend to prefer a freestanding machine over a NAS if only because the former tends to have much more computing power and will not bog down with a larger library as often.

    Next is selection of a server application platform or platforms or corollary application services, to access and manage the data and serve it to your clients. The primary ones are iTunes, DLNA, and some third-party music server apps, such as Roon and Logitech Media Server (LMS) (which is built for Squeezeboxes, but works with Chromecast clients). There are others. You can run multiple server applications accessing the same data! So if you are heavily invested in the Apple ecosystem, you can target your library into iTunes, and also set up a DLNA service for other non-Apple clients. Just be sure to not let iTunes or any other application make so many changes to the files and directories that it becomes difficult for the other server applications. By example, portions of my audio library are simultaneously accessed by both iTunes and LMS. I do NOT allow iTunes to "manage my library".

    While some of your clients can be connected via Wi-Fi, always keep your server machine and the data on a wired ethernet connection to your network. The latter may be asked to serve multiple streams to multiple clients simultaneously, and a gigabit ethernet link will provide the necessary bandwidth to do that. A Wi-Fi connection, even if AC, may not.

    Next comes the container and format of your data so that it is compatible with your server applications, clients, and at the quality level you want. That can become a bit of a juggling act. For instance, while many clients are fine playing FLAC files, iTunes and Apple clients are generally not. And while iTunes and Apple clients love Apple Lossless, many other non-Apple clients do not. You need to keep your files in a container format that each client can read and decode. Sometimes, you may need to create duplicate files in separate directories to accommodate the inconsistencies. For instance, many Apple mobile clients cannot handle 24-bit audio, but your other home clients may, and you may want to listen on both. As a default, nearly everything can play MP3 today, but that is a lossy low-fidelity format, IMO. WAV is another fairly universal format, and is what I have always ripped my CDs into beginning in 2001 - uncompressed 44.1/16 WAV. At that time, there were tagging and storage limitations, and a 250GB HDD was considered enormous. I did it anyway, because I remembered when a 20MB HDD was enormous. In 2019, there is no storage or streaming issue with uncompressed WAV, but it still does have limited tagging options. SO you may still want to make duplicates in an Apple or other format. But bear in mind that the best music server applications today do not require tags as much anymore, and can readily locate track, artist and other information online solely from the song titles. We did not enjoy that luxury in 2002. And for HD audio, always pick a format that will not exceed the wireless bandwidth of your clients. A lossless compressed one. For instance, do not stream 192/24 bit audio in a fully uncompressed format, as it will nearly saturate any client on a legacy G band. I usually keep my HD audio masters in FLAC, and will convert a copy to 44/16 Apple Lossless for Apple clients.

    How you tag your files is up to you. If you elect to use a container format that is good for tagging, you can add all sorts of information and graphics to each file. With WAV, if you toss a jpeg of the album cover in the same folder named as "folder", some server programs will apply it to the entire album. Other more powerful programs such as Roon will find everything for you as it indexes your library.

    For each server application or service, you then need to direct the applicable clients to them. It is relatively easy for Apple clients with iTunes and for the clients that are native to some third-party apps. DLNA is an open standard, and many clients not bound by their own proprietary server architecture will use that. I like LMS because I have Squeezeboxes everywhere, and it also serves my Chromecast clients well. But many new clients are 'Room-Ready' and are easy to interface into that ecosystem.

    The general rule of thumb is one server per client. Otherwise with some clients, each server application may may settings changes, and network mayhem will result.

    If you have dissimilar clients and multiple server applications and services, it is probably a good idea to have a powerful audio transcoding application, for conversions between formats on an ongoing basis. I recommend DB Poweramp, which has excellent flexibility and fast conversion with plenty of options into most common containers at varying bit and sample rates. It also includes an excellent ripping program. Other excellent CD ripping programs are Exact Audio Copy (PC) and XLD (Mac). XLD also has some conversion capability.

    That's a long initial guide on how to do this, in addition to the others that I'm sure are available here. Hope it helps.
     
    KeninDC and awillia6 like this.
  6. cpt_paranoia

    cpt_paranoia Super Member

    Messages:
    1,061
    I think that is overkill for data that changes infrequently, and is not 'mission critical'.

    You need backups, preferably more than one, and in physically disparate locations.

    RAID won't protect against fire, flood or theft. Multiple, physically separate backups will. If your primary disk fails, buy a new one, and clone one of the backups. You can start using one of the backup disk immediately, should you feel the need, and are prepared to risk the potential for further failures before you have restored your full backup set.

    Three disks, with two unpowered will be cheaper, more resilient to other loss mechanisms, take less power, and have a longer life than a RAID array.
     
    RTally and awillia6 like this.

     

    Please register to disable this ad.

  7. sgmlaw

    sgmlaw Well-Known Member

    Messages:
    651
    If you read the post (and it was admittedly long), backups were addressed as well. Including for on site catastrophe events by way of off-site backups.

    You want to avoid accessing your backups unless absolutely nesessary for recovery. A fault-tolerant primary storage system is a further form of backup in the event of drive failure. You can never have too many backups, but multiple single drive backups are difficult to maintain and combersome to synchronize data on. Having an unmounted backup volume is inconvenient for a dataset as dynamic as this. You are always adding new files to an audio file library and frequently modifying existing files over time. Updated and added files tend to be lost when backup volumes are unmounted, especially multiple ones under your scheme.

    I have maintained a continuously running RAID storage array for over 25 years. The drives become obsolete or reach capacity before they reach their failure window.
     
    Bill Ferris and awillia6 like this.
  8. cpt_paranoia

    cpt_paranoia Super Member

    Messages:
    1,061
    My data isn't that dynamic. Yes, I buy and rip CDs, a few a week, but each rip takes only 5 minutes or so, so the incremental effort to re-rip a small number of CDs in the event of the loss of my primary disk isn't that big a deal. If I was that bothered, I'd create an increment update on a drive on my PC (e.g. using xcopy), and use that to protect against loss of my primary drives between full incremental backup to my backup drive.

    I did say 'I think...'; it's just my opinion. We have different approaches to preserving our hard-won data. But, fundamentally, we have backups; that's the real takeaway message.
     
  9. awillia6

    awillia6 Super Member

    Messages:
    1,379
    That's where Apple's Time Machine utility (full/incremental backup/restore functionality fully integrated into the Mac operating system) is invaluable. I recently needed a photo I was sure had been stored in my photo library. Nope. Then I remembered I got that pic when someone emailed it to me and had never saved it to my photo library. So I fired up Time Machine and went back to recover the specific email that had the specific photo attached that I was missing. In less than 10 minutes I had my pic. TM works in the background automatically with no complicated setup, maintenance, or forced (compromised) choice of strategies. Just works.
     
    Last edited: Jan 8, 2019
  10. cpt_paranoia

    cpt_paranoia Super Member

    Messages:
    1,061
    I like to be in control of my data. I've seen too many reports of Time Machine 'just not working'...

    And I don't have a Mac. I don't use the (poor) tools provided by WD for use with my MyCloud NAS, preferring to use 3rd party tools that I can control, test, and are regularly updated, and not vendor-specific. The WD MyCloud forum is full of reports of the WD tools 'just not working', and their Time Machine server also 'just not working'...

    Understanding of, and confidence in, your backup tools is critical. I'm fortunate that my technical background means I can create my own (automated) tools without much effort. I have a script that synchronises selected folders on my PC with a backup on my NAS, launched from a desktop shortcut, that runs and then shuts down the pc. I use it whenever I've finished using the pc. This was previously based on xcopy, but now uses FreeFileSync, with a specific configuration file, to mirror or sync files as I require.
     
    awillia6 likes this.
  11. sgmlaw

    sgmlaw Well-Known Member

    Messages:
    651
    I hear you! I run TM, too, but do not exclusively rely on it. Very buggy. For machine backup, I also run Carbon Copy Cloner simultaneously, which is much more dependable. For continuous cloud backup, I prefer Backblaze, which has no data limit and good encryption. This provides good off-site protection.

    But for my audio and video library, while the working copies are automatically backed up to the cloud, I back it up manually to the network backup volume as the files are prepared, tagged and indexed. That makes the backup really easy to access in a pinch, no TM or volume mounting software required. Huge sparsebundles can be a PITA to work with. I don’t care for automatic mirroring to a local backup volume, if only to confirm working file changes are successful before they are sync’ed.

    All combined with audio and video, I think we are just past 12TB total. The only time we spin an optical disc now is long enough to rip it onto HDD.
     
    Last edited: Jan 8, 2019

     

    Please register to disable this ad.

  12. sgmlaw

    sgmlaw Well-Known Member

    Messages:
    651
    TM has outstanding versioning capabilities. And is the best first option to restore a machine. But it can get buggy over time as its sparsebundle grows and grows. I find after a year to 18 months I have to delete the sparsebundle and start over. That’s where the secondary and off-site backups give added comfort. If the machine crashes during the initial TM backup, you have a major problem
     
    awillia6 likes this.
  13. E-Stat

    E-Stat AK Subscriber Subscriber

    Messages:
    6,984
    I couldn't agree more with virtually every point you've made.

    RAID is for high transaction activity systems, not largely static databases. Keep the spares offline for improved protection and greater life.

    I run an SSD for the OS and image it using Macrium software following each monthly MS update. Have a spare and can restore an image in twenty mintues if the primary fails.

    All my data resides on a single 2.5" 2 TB drive. I keep half a dozen offline USB hard drives as backups. Most live in a fireproof safe while one lives at my wife's parent's house.
     
    Last edited: Jan 9, 2019
    Bill Ferris and awillia6 like this.
  14. E-Stat

    E-Stat AK Subscriber Subscriber

    Messages:
    6,984
    Always? I average about four albums a month. That's a five minute incremental backup performed on a weekly basis. Modifying existing ones? If I don't like the initial tagging, I correct that on the front end.

    The only files that change for me on a frequent basis are for my financial software where I maintain daily backups of about 1 GB on USB thumb drives. I find no need to spin a bunch of drives 24/7 for that purpose.
     
  15. sgmlaw

    sgmlaw Well-Known Member

    Messages:
    651
    . . . and you run to the fire safe to grab the other six USB hard drives, and then drive to your in-laws to get that drive, every time you get a new album? I think my setup is far more convenient. Three second drag and drop, and it’s done. Different strokes for different folks.

    RAID is not just for high transaction activity systems. It’s for data that you want to keep and keep on-line through a drive failure. That’s why every NAS appliance sold today uses it, even the cheaper consumer ones.
     
  16. E-Stat

    E-Stat AK Subscriber Subscriber

    Messages:
    6,984
    Run? I walk about thirty feet once a week to retrieve and update one of the drives at a time.

    Why would I do that? If you had a fire that leveled your house, the NAS would be toast.

    That reminds me of Brian Regan's take on those who microwave their Pop Tarts. ;)



    I've yet to suffer a drive failure in over thirty years. Don't know about you, but things have changed since the early 80s when I was working with 14" removable disk packs. I end up replacing them after several years as cost goes down and capacity up. I simply don't run any applications where a couple of minutes makes any difference. As for the customers running high transaction businesses I support, that's a different story. We prefer RAID 10 for their servers.

    Nothing succeeds like excess!
     
    Last edited: Jan 9, 2019

     

    Please register to disable this ad.

  17. cpt_paranoia

    cpt_paranoia Super Member

    Messages:
    1,061
    Plenty of consumer NAS drives don't use any form of RAID for user data; my (single HDD) WD MyCloud doesn't. There are two RAID partitions, but they're used for the system, not for user data. Synology also make single HDD, non RAID NAS drives.
     
  18. sgmlaw

    sgmlaw Well-Known Member

    Messages:
    651
    Correction: Every serious NAS appliance sold today uses RAID.

    A single drive solution, no matter how you partition it, is a data loss waiting to happen.
     
    Bill Ferris likes this.
  19. sgmlaw

    sgmlaw Well-Known Member

    Messages:
    651
    QED. And you would suffer a partial or total data loss with that practice, unless your in-laws’ single-drive backup was up-to-date.

    I just send a message to Backblaze, and all my data arrives on replacement drives the next morning. All of it. Because Backblaze keeps my data on their RAID modules. Just like any other responsible data center.

    RAID10 is a too expensive protocol to implement for homes and most smaller businesses. RAID5 represents a more acceptable cost to protection balance in those environments. And it is what most 4 or more drive consumer/prosumer NAS solutions implement by default. Combined with regular scheduled RAID scrubbings and a reliable automatic off-site solution, the statistical probability of data loss drops to an infinitesimal level.

    Anyone not experiencing an HDD failure in 30 years has either not worked with enough HDDs or is a very lucky aberration. I have personally witnessed over a dozen drive failures in data storage settings. Seagates in particular (even their so called enterprise grade line ) were notorious for burning up about 10 years ago. Every single time, the array redundancy protected the data.

    Excess? You’ll wish you had some excess if that one remaining single drive backup at the in-laws doesn’t spool up. I have to lose four HDDs simultaneously before we need to resort to off-site backups. Excluding site catastrophe events, that is statistically almost impossible.

    And from what you have explained, I am only using one more drive than you are to do this. One.

    I’ll reiterate. With unique data or data that represents significant hours of work time, you can never have too many backups or layers of protection. My time is too valuable to be stingy about protecting data.

    We centralize everything on our data arrays, including on them every file, document, photo, and E-mail we have created, sent or received since the early 1990s.

    _________________

    For the OP with 20,000 audio tracks, if they are not HD sized files, a reliable two-drive RAID1 box is around $2-300, and will provide good, fool-proof redundancy, and capacity headroom as you grow. If you want more expandability, you can move up to a 4-bay model, and only start by populating it with 2 drives, expanding later as your library expands.

    Attached directly to network via ethernet, it should be able to serve about 10-12 audio clients before saturating its interface bus. Less if it is asked to do more or other things, depending on the processing power of the box. It may be more responsive if connected directly to a server machine via USB3 or TB if that is the case.

    Whether to backup is a personal choice. Many of these appliances can be backed up natively through a cloud service, or if it is connected to a server machine via USB3 or TB, via that machine's backup systems. Or you can back it up manually.
     
    Last edited: Jan 10, 2019 at 10:18 AM
  20. rwartner

    rwartner Super Member

    Messages:
    1,189
    Location:
    Scottsdale, AZ
    I have laptop dedicated music player. I connect 2TB USB drive to it that contains 2700 CDs ripped lossless. I rip CDs on different laptop (this one that I am using now). I then copy to backup USB drive and then copy to USB drive on music laptop.
    The old sneaker network.
     
    Bill Ferris likes this.

Share This Page