Update: Microsoft finally released a tool to do what O365Migrator does! 🙂
When you’re moving from a homedirectories / fileshares (mapped to driveletters) based environment to a situation where you want your users to work in Onedrive for Business and Sharepoint Online, the best way to do so is to remove the homedirectories and shares and give your users access to Onedrive for Business and Sharepoint Online. One choice enforces adoption.
Of course, you also need to make the data compliant first, and upload it to Office 365. You can either let your users do this, or you can use my free tool!
O365Migrator will provision, authorize, clean up and upload. Everything you need to move from homedirectories or fileshares to Onedrive for Business and Sharepoint Online in one simple free tool.
Features
- pre-creates Onedrive Storage on demand for O365 licensed users
- sets administrative permissions on each account for the account used to migrate the data
- makes the homedirectories fully compliant with Onedrive requirements (no sync issues!)
- makes fileshares fully compliant with Sharepoint Online requirements
- automatic detection of the O4B url’s and library names
- multi-threading allows you to upload homedirectories or subfolders simultaneously
- full logging, automatic handling of errors where possible
- it’s powershell, open source, but it has a GUI!
- can be run on multiple machines or locations with different accounts at the same time
- retains original file creation dates
- Differential upload: only upload new/changed files
- Subfolder targeting
- Retry function for failed files/folders
- Your data is now safe from ransomware like Cryptolockers
- html report in an automatically created O4BMoved folder of any changes that had to be made to his/her folder/file structure
How to use for homedirectories
- Make sure your users exist in Office 365 and are licensed, several days before you migrate
- Ensure your admin/migration user has a license (E3/E1) or provisioning will fail
- Use the pre-provisioning and authorization options of O365Migrator to ensure their Onedrive storage space is created (this won’t overwrite any existing data)
- Ensure their onedrives have been provisioned (log in as a few of them or browse their unique url)
- identify the login names of the user you wish to migrate in UPN format (office 365 login name)
- identify the paths to their home directories (UNC or local)
- Create a CSV file with two columns. The header of the first colum should be ‘username’, the second column should be ‘path’. Then add your users to it.
- Verify the homedirectories
- Upload them
How to use for fileshares
- Identify the URL to your site (usually in the form of https://tenant.sharepoint.com/siteXXX
- Identify the document library’s title (the ‘name’ field in the library properties)
- Select the folder you wish to upload
- Verify it first with the left button
- Upload it with the right one
How to use for both of the above
- Install the required components on your migration machine:
- Sharepoint Online Client SDK MSI
- Sharepoint Online Management Module
- Powershell 4 / .NET 4.5
- Office 365 Online Module (only step 1Â is required)
- Run ‘Set-ExecutionPolicy Unrestricted’ in an elevated Powershell window, sign the script, or run the script in ByPass mode
- Download (O365Migrator_v1.00.zip) or O365Migrator_v0.9 and unzip it locally
- Right click all files and unblock them if necessary
- Run O365Migrator.ps1 from the cmdline, NOT from ISE! (leave the other files in the same folder!)
- The csv file in the zip is an example file for homedirect migration
- Log files will be in %APPDATA%
- Specifying a subfolder is optional, this will create a subfolder in the destination and upload there instead of to the root
- Make sure you don’t set the document library property that enforces checkout/checking
- Use a cloud-sourced Admin account (not synced from somewhere)
- Do not use MFA on this admin account, SpO CSOM doesn’t support MFA
Retrying failed items
The script has no built in functionality for this, you’ll have to either re-run it (note it will overwrite anything already there), or fix any issues you encounter in the log manually.
Speed considerations
Microsoft caps the maximum speed at which you can upload using CSOM. There are different throttles, the first is user based, which you can circumvent by using multiple instances. If you surpass that, you will get blocked at the tenant level. This is why you should always do a test migration first! If this tool can’t help you, use the Sharepoint Online File Import Service, which isn’t capped.
Important disclaimer
I’m not a real programmer, this code is meant as a Proof Of Concept. I do not guarantee this product will work in your setup, and I offer no dedicated support, I try to help everyone on a best-effort basis but also have to work for a living. So make sure you test well and understand the code before you use it.
Jos! Do you wear a cape? You should, because you are a hero! Where Microsoft epicly fails to provide tools and support to migrate from a fileshare to sharepoint/onedrive and other companies jump in with overpriced and overfunctional software you come up with this to save the day!
I almost gave up on moving a 200gb file share to sharepoint and tell the client to use other online storage.
Keep up the good work! Is there a donate button somewhere?
This is a fantastic and must have tool for any IT professional and/or small/medium IT business that regularly deals with uploading data to Office 365 platform! I have already uploaded over 250GB of data to around 100-125 OD4B sites and am in the process of migrating a large batch of data to SharePoint Online document library now. You can also learn a lot about PowerShell and CSOM if you take the time to learn the code provided.
Thank you for writing and sharing this for free guys! It’s worked perfectly for me to provision one drive for my domain users and move their files from file server to the cloud. Smaller batches have been done for testing , the rest are transferring now I’m confident enough, I couldnt believe how hassle free this tool was.
Top job guys , thanks for saving me a lot of time with this great tool.
Hello Jos, does your tool also detects references to unc paths in xls and doc files? I’m looking for this functionality for our file migration. Regards,
Ronald
Hi, I have started an upload for one user and its stuck on Uploading all folders from the CSV?
Love this script thank you. I noticed when it runs for Homedir migration there is a list of files it wont upload, is there a way to add directories to that?
Does this delete files that exists in Sharepoint but no longer exist from the source shared folder? similar to robobopy’s /purge switch..
basically i did an intial copy of the shared folder and it took a few days, so now i have to do another copy which will only copy new or changed files. but will it delete deleted files as well ?
Hello, I have an OnPrem only SharePoint installation. Could PowerShell be used to migrate from a file share to an on-prem SP Server and still retain the file properties, permissions, etc?
[…] Haven't used the tool but there is this from the same guy who made the OneDriveMapper script O365Migrator | Liebensraum […]
When you click provision is it doing it for all users? I hope not, we have 70000!
[…] worked very well so far. Another useful tool that we used for our DFS to SharePoint migration was O365Migrator, This PowerShell tool was a massive help and allowed us to bulk upload our DFS directories to […]
[…] are looking for this: O365Migrator | Liebensraum and this OneDriveMapper | […]
a tool for sharepoint migration from one tenant to another would be super useful for our primary schools to become part of our overarching tenant.
[…] O365 Migrator: https://www.lieben.nu/liebensraum/o365migrator/ […]
Hello, i’m using the tool to migratie some users to OneDrive. But there is one user who has a different site URL, in stead of https://contoso-my.sharepoint.com/personal/a_bcdefg_contoso_com/ it has a 1 behind the com so this will be https://contoso-my.sharepoint.com/personal/a_bcdefg_contoso_com1/
The migrator will not recognize this, and will fail with the message that the URL is not reachable or provisioned. Is this solvable within the Powershell script?
I keep receiving the following errors, any assistance would be much appriciated:
\\| VERBOSE | | | Starting v1.13 scan in Read-Write Mode | 12/11/2017 20:06:41
\\ | VERBOSE | | | Folder Ignore List Items: Non-Migrateable,Application Data,My Oce Printer Driver Templates,Windows,Remote Assistance Logs |\\ | CRITICAL ERROR | | | Failed to write Non-Migrateable folder | This operation is supported only when you are connected to the server.
I am using my domain admin creds.
[…] late, I have been using an excellent free tool called O365 Migrator which you can find here It has a host of features, including home drive migration to OneDrive for business, file shares to […]
many thanks, really a great resource and so much better than the script I created.
Great tool. Thanks for sharing. One recommendation is to update the language when you click upload as it implies that if the file exists it will be overwritten but as I understand it, it now does a differential copy.
Love the tool, but after struggling for 3 days of getting this to work, maybe it’s good to mention that you need to use a cloud only global admin account with MFA disabled otherwise things just won’t work. But now it’s uploading like a “malle” so thanks!!
I’m using the .9 download. I’ve done the Provision and Authorize steps with no errors a couple time. I also know I have full rights to the data source. The tool works with my migration but when trying the fist user I get: Failed to create new folder: Exception calling “ExecuteQuery” with “0” argument(s): “Access denied. You do not have permission to perform this action or access this resource.” Cannot continue because “the specified OneDrive subfolder could not be created
Any ideas???
[…] O365Migrator […]
I ran into numerous occasions where SaveBinaryDirect was failing for large files (> ~300MB) I have written another version of the Upload-File function which does chunked uploads, and so far it seems to be working well. It also appears there was a bug in the Upload-File function where it checks if the file already exists. A comparison is made between LastWriteTimeUtc and TimeCreated, when I believe the comparison should be between LastWriteTimeUtc and TimeLastModified. Here is a link to my updated uploader.ps1 if you’re interested in taking a look.
https://oregontech-my.sharepoint.com/personal/alan_wallace_oit_edu/_layouts/15/guestaccess.aspx?docid=095e9aa89f19841c9a6816709479b9364&authkey=Af2Z-gQ2hLXU8dgUktqdNAM
I think there may be a small bug in the cleaner. It appears that it replaces _t and _w if the folder name STARTS WITH those characters (cleaner.ps1, cleaner_sp.ps1, cleanerNoCommit.ps1 and cleaner_spNoCommit line 157) According to the OneDrive restrictions, the folder/ name cannot be EQUAL TO _w or _t, but it is valid for them to start with those characters. I have tested creating folders starting with those, and it handles them just fine.
This tool is AMAZING and has saved us countless hours and dollars moving our folders to OneDrive. Thanks for sharing with the world!
This looks like a great tool but Im interested to know how this connects to O365. Is some kind of cloud connector used? Do ports need to be opened on home drive servers. I can see the IT security team asking these kinds of questions.
Hi Jos,
First of all, thanks for creating the tool! However, i’ve run in some issues while using it.
Whenever I try to authenticate in the GUI my request times out, however, if I enter wrong credentials I get
Authentication Error: Bad username or password., zo i know my credentials are fine 🙂Hello, I would like to know when a file is skipped or uploaded in OneDrive. We have run the same batch twice and the reason why files are skipped is not clear.
It just works! Thank you very much!
Hi,
I keep getting these errors when uploading documents to my library:
Exception calling “SaveBinaryDirect” with “4” argument(s): “The remote server returned an error (500) internal server error”
Why?:(
No errors are detected with the “Scan and fix” button.
I’m running the v.99 version.
Thanks for this wonderful program which has been working fine – but now no matter whatever I do I keep getting a login failure ‘unexpected authentication failure’ any ideas?
Is it possible to exclude file extension types during the upload with this tool, without having to move them to an exclusion folder (folderIgnoreList)??
Thanks.
Hey Jos,
Thanks for this amazing tool. Been using it quite a bit lately and still have plenty to do. Have a question regarding the syntax in the “subfolder” name box on the folder migration tab. I have a library named Accounting Documents where the following exists at the root: Billing2016. I need to import into “2016” folder there. I’ve tried several variants in the “subfolder” box on the folder migration tab and not yet been successful (Billing2016, Billing2016, etc.). How do I need to enter this in the “subfolder” box?
Thanks
I think provisioning has changed, and doesn’t work in this script any more (for me anyway!)
Using:
Request-SPOPersonalSite -UserEmails $usersToProvision;
Instead of:
CreatePersonalSiteEnqueueBulk seems to work.
Hope this helps someone!
hi
when i try and tun version .99 im getting the following error in the powershell script – im just wondering if this is anything to worry about?
Version check: FAILED: O365Migrator version mismatch, current version: v0.99, latest version: v0.98
[…] O365Migrator […]
Hi I am trying to test this out and in the log I am getting the below error for the one test user I am trying and on the PS window I am getting “Cannot find an overload for “Load”and the argument count: “1” The user’s one drive is provisioned and I can login into it and see it ready. Thanks inadvance F:homeUserX | VERBOSE | Source folder exists | 05/25/2017 08:00:01 F:homeJnoone | VERBOSE | Starting final verification | 05/25/2017 08:00:01 F:homeJnoone | VERBOSE | Starting upload | 05/25/2017 08:00:05 F:homeJnoone | CRITICAL ERROR | Could not connect to… Read more »
Hi Jos, first of all – thank you in advance for a potentially great tool! Having a problem with Folder Migration function. After all fields been filled in I am clicking in Scan & Fix (or Scan only – the result is the same) and then Upload. First pop up windows comes with the general Warning about upload and second one said: “You need to verify the folder before uploading it”. Then it is not doing anything. I guess that valid for v0.9 of the tool as here is no “Verify” button at all in 0.98 (your instructions above are… Read more »
[…] O365Migrator […]
Hi,
Are there any known limits on file sizes? I thought OneDrive supported 10GB but I get the following for files over 500mb:
Exception calling “SaveBinaryDirect” with “4” argument(s): “The remote server returned an error: (404) Not Found.”
Thanks.
Hello,
Just wondering if others can post what sort of upload speed they’re getting? And if there is any way to speed it up?
I’m currently uploading 40GB with 400Mbps of bandwidth and an estimated time for 10hours.
It’s currently afterhours where I am, so I should have pretty much the full 400Mbps available to me. Obviously there are limitations with Microsoft’s end as well.
Thanks.
[…] O365Migrator […]
Hi Jos, Great work on the script but I’ve run into a few issues with the Data Cleaner side. I firstly tried to use 0.97. When I run Scan and Fix, it displays “Method invocation failed because [Alphaleonis.Win32.Filesystem.FileInfo] does not contain a method named ‘new'” in the CleanerResults for any files that it tries to rename, i.e. long file names or files with incorrect characters. I then tried to use 0.9. I had more success with this, as it truncated long file and folder names, however, it was unable to change files with the following characters in the name ,… Read more »
[…] O365Migrator […]
After a amout of time the tool seems to hang. Is there a solution for this problem?
I uploaded 3 GB of data but then it stops uploading.
Hey Jos, Once again, thanks for another tool which is helping your fellow professionals around the world!! Does the latest O365Migrator or any possible future iterations have the ability to re-import data prior to a particular timestamp? For example in my dilemma I used your tool to import data to Sharepoint but the ‘date modified’ for folders and files show as the date that I migrated the data. Would be really cool if the tool allowed us to import data before a certain time so that we can reimport data to fix the modified data of files and folders without… Read more »
Hi, thanks for putting together a great script. I was just wondering if it is possible to avoid having a O4BMoved folder and the MovedFilesAndFoldersReport.html written to each users OneDrive?
I keep getting errors for the .retry files, here is what I see when trying to add a retry file:
: Retry file specified: C:Users…O365MigratorFailedUploadObjectsForRetry….retry
: This only has effect on the Folder Migration Tab. If you wish to use a retry file for a homedirectory, process each homedirectory individually on this tab.
: To unload the retry file and run normally, restart O365Migrator
I have tried restarting the process many times with no success. Should I be doing this differently?
[…] O365Migrator […]
I get the error CSV file invalid.. you need to insert a valid CSV first. But i used the example, with a change to my users. Whats going wrong?
Really great tool for migrating users files, would be really interested in the permissions tool if it becomes available
Hi Jos,
Really loved it.. Can the tool migrate the permissions as well ?
Hi Jos, thanks for the effort – very cool!
My scenario is: do a first pass, then, a week later do a second “delta” pass.
It may be that files are changed and maybe deleted between the first and second pass.
Can the tool detect and handle deletions in this scenario?
Many thanks.
I did not found any activities in log that when sync is started and when sync is completed. Is this possible?
Since running this application the account used has been given full ownership of all files and folders across 365 tenant. Do you have a method to remove this functionality once data migrations have been completed?
Super Tool, I Use it a lot. I encounter the following things in migrating to SharePoint with this tool. 1. The forbidden character “~” is also not allowed on SharePoint this character isn’t scanned by the O365Datacleaner. 2. Could you also make a button “Scan Only” instead of “Scan and Fix” this wil allow us to make corrections in case the file is nested to deep and moved to O4BMoved, if many files are nested to deep from a lot of differtent folders it is much work to place them again in the right place. You have to work through… Read more »
The delta sync seems very slow, which seems to be because it compares file by file and then skips if it is identical, so does not save much time over a full sync. Could you not extract a file list attribute set from O365 and compare to the local home directory and generate a sync delta of changed files or similar to help speed up the process? Ideally you want to do an initial pass to load the files, and then a final sync a few days later when the user is ready to cut over, but this should be… Read more »
I’m getting constant throttling errors (429) when combining your script with other software, is there a way to avoid this?
Hi, thanks for a great tool. It works flawless for homedirs which reside on the same server as the O365 migratot, However when I have homedirectories on another server I keep getting the error that the homedirectories can not be accessed. Any ideas on how to work around this?
Thanks
[…] O365Migrator will provision, authorize, clean up and upload. Everything you need to move from homedirectories or fileshares to Onedrive for Business and Sharepoint Online in one simple free tool. https://www.lieben.nu/liebensraum/o365migrator […]
Hello, thanks for this wonderful tool. Will this work on a federated O365 scenario? While I am logging into my federated domain using the migrator tool, it’s giving me an error “Authentication Error: Unexpected authentication failure”.
Has there been anything done about the changes to the URL of the OD4B sites from March 2017. You can no longer use .onmicrosoft.com to connect to a site. It now uses a different format (basically just your domain name).