Battling Big Files

I’ve just returned from a snowy Amsterdam where I attended and presented at the International Digital Curation Conference. Although a former colleague of mine (Ed Bremner) was there to take photos and record the plenary speakers he was unable to clone himself, so I helped video some of the parallel talks using a Flip camera. On the day I battled with the Flip battery life and storage space – Flips only have a 100 minutes of space and need frequent recharging – but I managed to record 12 videos. The videos vary in size but most are around 1 GB.

Since then I have battled with uploading the files to somewhere Kirsty Pitkin (event amplifier) can access them – she is going to edit the files. The place could be public but mustn’t be advertised – so YouTube and similar video sharing sites are out.

So I’m sat at the end of a snow-covered home broadband line with speed restrictions and file upload limits. The whole thing has been a painful process and I think I’ve been going about it all wrong. I am having my own big data problem – ironic in that I advise people on research data management! I’m sure there must be an easier way…any ideas please say!!

Too big!

The approaches I’ve tried/considered so far are:

  • Emailing files was impossible as they are just too big.
  • Uploading the files to my public area in Dropbox – this seems to be the most successful approach but is very slow and storage is limited by having a free account. Some files are too big to upload via the Web site so have had to be added using the file system.
  • Uploading the files to a public area of my personal work server. This approach requires VPN and breaks in my connection mean I have sent back to square one a few times.
  • Putting the files onto memory sticks or a disc and posting it – I didn’t try this as I was hoping using an online approach would be quicker – I may have been wrong, crawling round to Kirsty’s house on my hands and knees may be quicker.
  • Uploading files to Microsoft Skydrive or Google Drive – again some files are too big and I was reluctant to try another approach when Dropbox seemed to be working. Each time I try something different I’m guessing it just puts a little more strain on the upload pipe?
  • Uploading the files to BitTorrent or other filesharing services might be one way to do it, but they can be a bit dodgy.
  • Whatever happened to FTP?!

As I write I have now uploaded 9 of the 12 videos, I’m hoping the others will be in the right place by this time next year… ☹


2 thoughts on “Battling Big Files

  1. We’ve been really feeling for you at this end of the process, Marieke, and appreciate your perseverance!

    We had hoped that the Flip videos would be smaller and therefore easier to transfer, but that is obviously not the case. Ed ended up bringing the plenary videos around to us on SD cards on his way back from the airport and sitting with us whilst we transferred it all, which seems to have been the quickest option.

    Your mention of YouTube and video sharing sites did trigger one thought, which is that it is possible to upload videos to Vimeo and set the privacy settings to allow only chosen people to view the videos. Unlike YouTube, there is also an official download option, so we could download them at this end.

    That said, I’m not sure that would make life any easier than Dropbox, if the problem is the speed of data transfer….

    Hope someone can suggest a magical speedy way to get the remaining three across and release you!

Comments are closed.