O365Migrator

Update: Microsoft finally released a tool to do what O365Migrator does! 🙂

When you’re moving from a homedirectories / fileshares (mapped to driveletters) based environment to a situation where you want your users to work in Onedrive for Business and Sharepoint Online, the best way to do so is to remove the homedirectories and shares and give your users access to Onedrive for Business and Sharepoint Online. One choice enforces adoption.

Of course, you also need to make the data compliant first, and upload it to Office 365. You can either let your users do this, or you can use my free tool!

O365Migrator will provision, authorize, clean up and upload. Everything you need to move from homedirectories or fileshares to Onedrive for Business and Sharepoint Online in one simple free tool.

O365Migrator_v0.8

Features

How to use for homedirectories

  1. Make sure your users exist in Office 365 and are licensed, several days before you migrate
  2. Ensure your admin/migration user has a license (E3/E1) or provisioning will fail
  3. Use the pre-provisioning and authorization options of O365Migrator to ensure their Onedrive storage space is created (this won’t overwrite any existing data)
  4. Ensure their onedrives have been provisioned (log in as a few of them or browse their unique url)
  5. identify the login names of the user you wish to migrate in UPN format (office 365 login name)
  6. identify the paths to their home directories (UNC or local)
  7. Create a CSV file with two columns. The header of the first colum should be ‘username’, the second column should be ‘path’. Then add your users to it.
  8. Verify the homedirectories
  9. Upload them

How to use for fileshares

  1. Identify the URL to your site (usually in the form of https://tenant.sharepoint.com/siteXXX
  2. Identify the document library’s title (the ‘name’ field in the library properties)
  3. Select the folder you wish to upload
  4. Verify it first with the left button
  5. Upload it with the right one

How to use for both of the above

  1. Install the required components on your migration machine:
    1. Sharepoint Online Client SDK MSI
    2. Sharepoint Online Management Module
    3. Powershell 4 / .NET 4.5
    4. Office 365 Online Module (only step 1 is required)
  2. Run ‘Set-ExecutionPolicy Unrestricted’ in an elevated Powershell window, sign the script, or run the script in ByPass mode
  3. Download (O365Migrator_v1.00.zip) or O365Migrator_v0.9 and unzip it locally
  4. Right click all files and unblock them if necessary
  5. Run O365Migrator.ps1 from the cmdline, NOT from ISE! (leave the other files in the same folder!)
  6. The csv file in the zip is an example file for homedirect migration
  7. Log files will be in %APPDATA%
  8. Specifying a subfolder is optional, this will create a subfolder in the destination and upload there instead of to the root
  9. Make sure you don’t set the document library property that enforces checkout/checking
  10. Use a cloud-sourced Admin account (not synced from somewhere)
  11. Do not use MFA on this admin account, SpO CSOM doesn’t support MFA

Retrying failed items

The script has no built in functionality for this, you’ll have to either re-run it (note it will overwrite anything already there), or fix any issues you encounter in the log manually.

Speed considerations

Microsoft caps the maximum speed at which you can upload using CSOM. There are different throttles, the first is user based, which you can circumvent by using multiple instances. If you surpass that, you will get blocked at the tenant level. This is why you should always do a test migration first! If this tool can’t help you, use the Sharepoint Online File Import Service, which isn’t capped.

Important disclaimer

I’m not a real programmer, this code is meant as a Proof Of Concept. I do not guarantee this product will work in your setup, and I offer no dedicated support, I try to help everyone on a best-effort basis but also have to work for a living. So make sure you test well and understand the code before you use it.

Leave a Reply

109 Comments on "O365Migrator"

Notify of
avatar
Sort by:   newest | oldest | most voted
Jamie
Guest

Thank you for writing and sharing this for free guys! It’s worked perfectly for me to provision one drive for my domain users and move their files from file server to the cloud. Smaller batches have been done for testing , the rest are transferring now I’m confident enough, I couldnt believe how hassle free this tool was.

Top job guys , thanks for saving me a lot of time with this great tool.

Sean
Guest

This is a fantastic and must have tool for any IT professional and/or small/medium IT business that regularly deals with uploading data to Office 365 platform! I have already uploaded over 250GB of data to around 100-125 OD4B sites and am in the process of migrating a large batch of data to SharePoint Online document library now. You can also learn a lot about PowerShell and CSOM if you take the time to learn the code provided.

Sven
Guest

Jos! Do you wear a cape? You should, because you are a hero! Where Microsoft epicly fails to provide tools and support to migrate from a fileshare to sharepoint/onedrive and other companies jump in with overpriced and overfunctional software you come up with this to save the day!

I almost gave up on moving a 200gb file share to sharepoint and tell the client to use other online storage.

Keep up the good work! Is there a donate button somewhere?

david
Guest

many thanks, really a great resource and so much better than the script I created.

Sean Hanna
Guest

Great tool. Thanks for sharing. One recommendation is to update the language when you click upload as it implies that if the file exists it will be overwritten but as I understand it, it now does a differential copy.

Edwin
Guest

Love the tool, but after struggling for 3 days of getting this to work, maybe it’s good to mention that you need to use a cloud only global admin account with MFA disabled otherwise things just won’t work. But now it’s uploading like a “malle” so thanks!!

Ken
Guest

I’m using the .9 download. I’ve done the Provision and Authorize steps with no errors a couple time. I also know I have full rights to the data source. The tool works with my migration but when trying the fist user I get: Failed to create new folder: Exception calling “ExecuteQuery” with “0” argument(s): “Access denied. You do not have permission to perform this action or access this resource.” Cannot continue because “the specified OneDrive subfolder could not be created
Any ideas???

trackback

[…] O365Migrator […]

Alan
Guest

I ran into numerous occasions where SaveBinaryDirect was failing for large files (> ~300MB) I have written another version of the Upload-File function which does chunked uploads, and so far it seems to be working well. It also appears there was a bug in the Upload-File function where it checks if the file already exists. A comparison is made between LastWriteTimeUtc and TimeCreated, when I believe the comparison should be between LastWriteTimeUtc and TimeLastModified. Here is a link to my updated uploader.ps1 if you’re interested in taking a look.
https://oregontech-my.sharepoint.com/personal/alan_wallace_oit_edu/_layouts/15/guestaccess.aspx?docid=095e9aa89f19841c9a6816709479b9364&authkey=Af2Z-gQ2hLXU8dgUktqdNAM

Alan
Guest

I think there may be a small bug in the cleaner. It appears that it replaces _t and _w if the folder name STARTS WITH those characters (cleaner.ps1, cleaner_sp.ps1, cleanerNoCommit.ps1 and cleaner_spNoCommit line 157) According to the OneDrive restrictions, the folder/ name cannot be EQUAL TO _w or _t, but it is valid for them to start with those characters. I have tested creating folders starting with those, and it handles them just fine.

This tool is AMAZING and has saved us countless hours and dollars moving our folders to OneDrive. Thanks for sharing with the world!

Paul
Guest

This looks like a great tool but Im interested to know how this connects to O365. Is some kind of cloud connector used? Do ports need to be opened on home drive servers. I can see the IT security team asking these kinds of questions.

chris
Guest

Hi Jos,
First of all, thanks for creating the tool! However, i’ve run in some issues while using it.

Whenever I try to authenticate in the GUI my request times out, however, if I enter wrong credentials I get Authentication Error: Bad username or password., zo i know my credentials are fine 🙂

Victor
Guest

Hello, I would like to know when a file is skipped or uploaded in OneDrive. We have run the same batch twice and the reason why files are skipped is not clear.

Andi
Guest

It just works! Thank you very much!

Jonatan
Guest

Hi,

I keep getting these errors when uploading documents to my library:
Exception calling “SaveBinaryDirect” with “4” argument(s): “The remote server returned an error (500) internal server error”

Why?:(

No errors are detected with the “Scan and fix” button.

I’m running the v.99 version.

AWade
Guest

Thanks for this wonderful program which has been working fine – but now no matter whatever I do I keep getting a login failure ‘unexpected authentication failure’ any ideas?

Jos
Guest
AWade
Guest
Thanks Jos. That didn’t work for me but eventually I have got round the problem like this. I’m not sure if all these steps are necessary or even why it works but I installed the sharepoint online management shell, ran it as admin and logged in successfully using connect-spoService. Next I changed the O368Migrator.ps1 file from Connect-MsolService to Connect-spoService, set execution policy to unrestricted and then ran the script. Once I enter my credentials in the UI I then have to enter the address of my sharepoint admin site in the url prompt that appear in the background Powershell box.… Read more »
David
Guest

Is it possible to exclude file extension types during the upload with this tool, without having to move them to an exclusion folder (folderIgnoreList)??

Thanks.

Jos
Guest

Not without a change to the code

Jeff
Guest
Hey Jos, Thanks for this amazing tool. Been using it quite a bit lately and still have plenty to do. Have a question regarding the syntax in the “subfolder” name box on the folder migration tab. I have a library named Accounting Documents where the following exists at the root: Billing2016. I need to import into “2016” folder there. I’ve tried several variants in the “subfolder” box on the folder migration tab and not yet been successful (Billing2016, Billing2016, etc.). How do I need to enter this in the “subfolder” box? Thanks
Jeff
Guest

Hmm.. that should be Bililng2016. Seems the website doesn’t like that.

Paul
Guest

I think provisioning has changed, and doesn’t work in this script any more (for me anyway!)
Using:
Request-SPOPersonalSite -UserEmails $usersToProvision;
Instead of:
CreatePersonalSiteEnqueueBulk seems to work.

Hope this helps someone!

Russell
Guest

hi
when i try and tun version .99 im getting the following error in the powershell script – im just wondering if this is anything to worry about?

Version check: FAILED: O365Migrator version mismatch, current version: v0.99, latest version: v0.98

trackback

[…] O365Migrator […]

Jeremy
Guest
Hi I am trying to test this out and in the log I am getting the below error for the one test user I am trying and on the PS window I am getting “Cannot find an overload for “Load”and the argument count: “1” The user’s one drive is provisioned and I can login into it and see it ready. Thanks inadvance F:homeUserX | VERBOSE | Source folder exists | 05/25/2017 08:00:01 F:homeJnoone | VERBOSE | Starting final verification | 05/25/2017 08:00:01 F:homeJnoone | VERBOSE | Starting upload | 05/25/2017 08:00:05 F:homeJnoone | CRITICAL ERROR | Could not connect to… Read more »
Alex.
Guest
Hi Jos, first of all – thank you in advance for a potentially great tool! Having a problem with Folder Migration function. After all fields been filled in I am clicking in Scan & Fix (or Scan only – the result is the same) and then Upload. First pop up windows comes with the general Warning about upload and second one said: “You need to verify the folder before uploading it”. Then it is not doing anything. I guess that valid for v0.9 of the tool as here is no “Verify” button at all in 0.98 (your instructions above are… Read more »
Alex.
Guest

Btw – 0.9 version works! The only question I have – is it possible to use nested folder in Subfolder name field? I mean I want to copy the data to Folder 3 which is located at Folder1/Folder2/Folder3?
Thanks again!

Matthew
Guest

Hi,
Are there any known limits on file sizes? I thought OneDrive supported 10GB but I get the following for files over 500mb:
Exception calling “SaveBinaryDirect” with “4” argument(s): “The remote server returned an error: (404) Not Found.”
Thanks.

Matthew
Guest

Hello,
Just wondering if others can post what sort of upload speed they’re getting? And if there is any way to speed it up?
I’m currently uploading 40GB with 400Mbps of bandwidth and an estimated time for 10hours.
It’s currently afterhours where I am, so I should have pretty much the full 400Mbps available to me. Obviously there are limitations with Microsoft’s end as well.
Thanks.

trackback

[…] O365Migrator […]

Matthew
Guest
Hi Jos, Great work on the script but I’ve run into a few issues with the Data Cleaner side. I firstly tried to use 0.97. When I run Scan and Fix, it displays “Method invocation failed because [Alphaleonis.Win32.Filesystem.FileInfo] does not contain a method named ‘new'” in the CleanerResults for any files that it tries to rename, i.e. long file names or files with incorrect characters. I then tried to use 0.9. I had more success with this, as it truncated long file and folder names, however, it was unable to change files with the following characters in the name ,… Read more »
trackback

[…] O365Migrator […]

Sander
Guest

After a amout of time the tool seems to hang. Is there a solution for this problem?
I uploaded 3 GB of data but then it stops uploading.

Gee
Guest
Hey Jos, Once again, thanks for another tool which is helping your fellow professionals around the world!! Does the latest O365Migrator or any possible future iterations have the ability to re-import data prior to a particular timestamp? For example in my dilemma I used your tool to import data to Sharepoint but the ‘date modified’ for folders and files show as the date that I migrated the data. Would be really cool if the tool allowed us to import data before a certain time so that we can reimport data to fix the modified data of files and folders without… Read more »
Johnny
Guest

Hi, thanks for putting together a great script. I was just wondering if it is possible to avoid having a O4BMoved folder and the MovedFilesAndFoldersReport.html written to each users OneDrive?

Parmar
Guest

I keep getting errors for the .retry files, here is what I see when trying to add a retry file:

: Retry file specified: C:Users…O365MigratorFailedUploadObjectsForRetry….retry
: This only has effect on the Folder Migration Tab. If you wish to use a retry file for a homedirectory, process each homedirectory individually on this tab.

: To unload the retry file and run normally, restart O365Migrator

I have tried restarting the process many times with no success. Should I be doing this differently?

trackback

[…] O365Migrator […]

Maurice
Guest

I get the error CSV file invalid.. you need to insert a valid CSV first. But i used the example, with a change to my users. Whats going wrong?

Phil warren
Guest

Really great tool for migrating users files, would be really interested in the permissions tool if it becomes available

Sourav
Guest

Hi Jos,
Really loved it.. Can the tool migrate the permissions as well ?

Doug
Guest

Hi Jos, thanks for the effort – very cool!

My scenario is: do a first pass, then, a week later do a second “delta” pass.

It may be that files are changed and maybe deleted between the first and second pass.

Can the tool detect and handle deletions in this scenario?

Many thanks.

Avian
Guest

I did not found any activities in log that when sync is started and when sync is completed. Is this possible?

Calvin
Guest

Since running this application the account used has been given full ownership of all files and folders across 365 tenant. Do you have a method to remove this functionality once data migrations have been completed?

Sander
Guest
Super Tool, I Use it a lot. I encounter the following things in migrating to SharePoint with this tool. 1. The forbidden character “~” is also not allowed on SharePoint this character isn’t scanned by the O365Datacleaner. 2. Could you also make a button “Scan Only” instead of “Scan and Fix” this wil allow us to make corrections in case the file is nested to deep and moved to O4BMoved, if many files are nested to deep from a lot of differtent folders it is much work to place them again in the right place. You have to work through… Read more »
Olly
Guest
The delta sync seems very slow, which seems to be because it compares file by file and then skips if it is identical, so does not save much time over a full sync. Could you not extract a file list attribute set from O365 and compare to the local home directory and generate a sync delta of changed files or similar to help speed up the process? Ideally you want to do an initial pass to load the files, and then a final sync a few days later when the user is ready to cut over, but this should be… Read more »
Rene
Guest

I’m getting constant throttling errors (429) when combining your script with other software, is there a way to avoid this?

Xander
Guest

Hi, thanks for a great tool. It works flawless for homedirs which reside on the same server as the O365 migratot, However when I have homedirectories on another server I keep getting the error that the homedirectories can not be accessed. Any ideas on how to work around this?
Thanks

trackback

[…] O365Migrator will provision, authorize, clean up and upload. Everything you need to move from homedirectories or fileshares to Onedrive for Business and Sharepoint Online in one simple free tool. http://www.lieben.nu/liebensraum/o365migrator […]

Richard Parry
Guest

There are a few folder I’d like to exclude from migration – favorites (it’s redirected) and anything in Recycle.bin

I can see references to folderIgnoreList and the folderignore array but I can’t find the list of excluded folders to add these folders to the list.

Richard Parry
Guest

Never mind, found it – 365Migrator.ps1

David
Guest

Hello, thanks for this wonderful tool. Will this work on a federated O365 scenario? While I am logging into my federated domain using the migrator tool, it’s giving me an error “Authentication Error: Unexpected authentication failure”.

Jeff
Guest

Has there been anything done about the changes to the URL of the OD4B sites from March 2017. You can no longer use .onmicrosoft.com to connect to a site. It now uses a different format (basically just your domain name).

wpDiscuz