Using OneDrive to sync development directories

Written by James McDonald

August 23, 2021

TLDR; Don’t use OneDrive to sync your development project directories

So I have a development projects directory that I wanted to keep backed up in case I lost my laptop. So I copied the entire thing into my OneDrive

I use CakePHP and React for programming mainly so you get massive vendor and node_modules directories in each project folder

What a nightmare using OneDrive to sync files – it’s incredibly patchy and bug ridden. You can’t easily sync the files between MacOS and Windows and Linux as the OneDrive client doesn’t perform on MacOS and doesn’t exist on Linux (I use Insync but meh) and you have to then selectively sync folders so that the MacOS and Linux clients don’t have too much to do

I think Dropbox for business is a way better product

How to lessen the pain

  1. Remove the vendor and node_modules folders to reduce what OneDrive is syncing
  2. Create a development folder outside of your onedrive tree and then use git to clone your onedrive git repos
  3. Commit your changes back to the onedrive hosted repositories on a schedule

Make a dev directory outside Onedrive

mkdir c:\dev
cd C:\dev
git clone C:\Users\Rupert\Onedrive\Sites\repo1
mkdir c:\dev\bin

Create a sync script to run as a scheduled task to commit your local changes automatically to onedrive sans the node_modules or vendor dirs (add them to .gitignore if not already)

#!/bin/sh

# put this in c:\dev\bin\onedrive-repo-backup.sh

COMMIT_DATE=`date +"%Y-%m-%d"`
cd /c/dev/get-current-crypto-aud
# git add . -A
# just commit what is already commited
git commit -m "Changes committed ${COMMIT_DATE}"
# push to the backup branch
git push onedrive master:backup

Create a scheduled task to call the above sync script for your repositories. Add to the code to sync multiple repos.

Removing a heap of folders from your local OneDrive directories programmatically

You might think that you just run remove-item recurse but it won’t remove the special NTFS link files that are “online only” pointers to the actual files stored in your onedrive so the following removed the vendor and node_modules folder

function Remove-FileSystemItem {
    <#
  .SYNOPSIS
    Removes files or directories reliably and synchronously.

  .DESCRIPTION
    Removes files and directories, ensuring reliable and synchronous
    behavior across all supported platforms.

    The syntax is a subset of what Remove-Item supports; notably,
    -Include / -Exclude and -Force are NOT supported; -Force is implied.
    
    As with Remove-Item, passing -Recurse is required to avoid a prompt when 
    deleting a non-empty directory.

    IMPORTANT:
      * On Unix platforms, this function is merely a wrapper for Remove-Item, 
        where the latter works reliably and synchronously, but on Windows a 
        custom implementation must be used to ensure reliable and synchronous 
        behavior. See https://github.com/PowerShell/PowerShell/issues/8211

    * On Windows:
      * The *parent directory* of a directory being removed must be 
        *writable* for the synchronous custom implementation to work.
      * The custom implementation is also applied when deleting 
         directories on *network drives*.

    * If an indefinitely *locked* file or directory is encountered, removal is aborted.
      By contrast, files opened with FILE_SHARE_DELETE / 
      [System.IO.FileShare]::Delete on Windows do NOT prevent removal, 
      though they do live on under a temporary name in the parent directory 
      until the last handle to them is closed.

    * Hidden files and files with the read-only attribute:
      * These are *quietly removed*; in other words: this function invariably
        behaves like `Remove-Item -Force`.
      * Note, however, that in order to target hidden files / directories
        as *input*, you must specify them as a *literal* path, because they
        won't be found via a wildcard expression.

    * The reliable custom implementation on Windows comes at the cost of
      decreased performance.

  .EXAMPLE
    Remove-FileSystemItem C:\tmp -Recurse

    Synchronously removes directory C:\tmp and all its content.
  #>
    [CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'Medium', DefaultParameterSetName = 'Path', PositionalBinding = $false)]
    param(
        [Parameter(ParameterSetName = 'Path', Mandatory, Position = 0, ValueFromPipeline, ValueFromPipelineByPropertyName)]
        [string[]] $Path
        ,
        [Parameter(ParameterSetName = 'Literalpath', ValueFromPipelineByPropertyName)]
        [Alias('PSPath')]
        [string[]] $LiteralPath
        ,
        [switch] $Recurse
    )
    begin {
        # !! Workaround for https://github.com/PowerShell/PowerShell/issues/1759
        if ($ErrorActionPreference -eq [System.Management.Automation.ActionPreference]::Ignore) { $ErrorActionPreference = 'Ignore' }
        $targetPath = ''
        $yesToAll = $noToAll = $false
        function trimTrailingPathSep([string] $itemPath) {
            if ($itemPath[-1] -in '\', '/') {
                # Trim the trailing separator, unless the path is a root path such as '/' or 'c:\'
                if ($itemPath.Length -gt 1 -and $itemPath -notmatch '^[^:\\/]+:.$') {
                    $itemPath = $itemPath.Substring(0, $itemPath.Length - 1)
                }
            }
            $itemPath
        }
        function getTempPathOnSameVolume([string] $itemPath, [string] $tempDir) {
            if (-not $tempDir) { $tempDir = [IO.Path]::GetDirectoryName($itemPath) }
            [IO.Path]::Combine($tempDir, [IO.Path]::GetRandomFileName())
        }
        function syncRemoveFile([string] $filePath, [string] $tempDir) {
            # Clear the ReadOnly attribute, if present.
            if (($attribs = [IO.File]::GetAttributes($filePath)) -band [System.IO.FileAttributes]::ReadOnly) {
                [IO.File]::SetAttributes($filePath, $attribs -band -bnot [System.IO.FileAttributes]::ReadOnly)
            }
            $tempPath = getTempPathOnSameVolume $filePath $tempDir
            [IO.File]::Move($filePath, $tempPath)
            [IO.File]::Delete($tempPath)
        }
        function syncRemoveDir([string] $dirPath, [switch] $recursing) {
            if (-not $recursing) { $dirPathParent = [IO.Path]::GetDirectoryName($dirPath) }
            # Clear the ReadOnly attribute, if present.
            # Note: [IO.File]::*Attributes() is also used for *directories*; [IO.Directory] doesn't have attribute-related methods.
            if (($attribs = [IO.File]::GetAttributes($dirPath)) -band [System.IO.FileAttributes]::ReadOnly) {
                [IO.File]::SetAttributes($dirPath, $attribs -band -bnot [System.IO.FileAttributes]::ReadOnly)
            }
            # Remove all children synchronously.
            $isFirstChild = $true
            foreach ($item in [IO.directory]::EnumerateFileSystemEntries($dirPath)) {
                if (-not $recursing -and -not $Recurse -and $isFirstChild) {
                    # If -Recurse wasn't specified, prompt for nonempty dirs.
                    $isFirstChild = $false
                    # Note: If -Confirm was also passed, this prompt is displayed *in addition*, after the standard $PSCmdlet.ShouldProcess() prompt.
                    #       While Remove-Item also prompts twice in this scenario, it shows the has-children prompt *first*.
                    if (-not $PSCmdlet.ShouldContinue("The item at '$dirPath' has children and the -Recurse switch was not specified. If you continue, all children will be removed with the item. Are you sure you want to continue?", 'Confirm', ([ref] $yesToAll), ([ref] $noToAll))) { return }
                }
                $itemPath = [IO.Path]::Combine($dirPath, $item)
                ([ref] $targetPath).Value = $itemPath
                if ([IO.Directory]::Exists($itemPath)) {
                    syncremoveDir $itemPath -recursing
                }
                else {
                    syncremoveFile $itemPath $dirPathParent
                }
            }
            # Finally, remove the directory itself synchronously.
            ([ref] $targetPath).Value = $dirPath
            $tempPath = getTempPathOnSameVolume $dirPath $dirPathParent
            [IO.Directory]::Move($dirPath, $tempPath)
            [IO.Directory]::Delete($tempPath)
        }
    }

    process {
        $isLiteral = $PSCmdlet.ParameterSetName -eq 'LiteralPath'
        if ($env:OS -ne 'Windows_NT') {
            # Unix: simply pass through to Remove-Item, which on Unix works reliably and synchronously
            Remove-Item @PSBoundParameters
        }
        else {
            # Windows: use synchronous custom implementation
            foreach ($rawPath in ($Path, $LiteralPath)[$isLiteral]) {
                # Resolve the paths to full, filesystem-native paths.
                try {
                    # !! Convert-Path does find hidden items via *literal* paths, but not via *wildcards* - and it has no -Force switch (yet)
                    # !! See https://github.com/PowerShell/PowerShell/issues/6501
                    $resolvedPaths = if ($isLiteral) { Convert-Path -ErrorAction Stop -LiteralPath $rawPath } else { Convert-Path -ErrorAction Stop -path $rawPath }
                }
                catch {
                    Write-Error $_ # relay error, but in the name of this function
                    continue
                }
                try {
                    $isDir = $false
                    foreach ($resolvedPath in $resolvedPaths) {
                        # -WhatIf and -Confirm support.
                        if (-not $PSCmdlet.ShouldProcess($resolvedPath)) { continue }
                        if ($isDir = [IO.Directory]::Exists($resolvedPath)) {
                            # dir.
                            # !! A trailing '\' or '/' causes directory removal to fail ("in use"), so we trim it first.
                            syncRemoveDir (trimTrailingPathSep $resolvedPath)
                        }
                        elseif ([IO.File]::Exists($resolvedPath)) {
                            # file
                            syncRemoveFile $resolvedPath
                        }
                        else {
                            Throw "Not a file-system path or no longer extant: $resolvedPath"
                        }
                    }
                }
                catch {
                    if ($isDir) {
                        $exc = $_.Exception
                        if ($exc.InnerException) { $exc = $exc.InnerException }
                        if ($targetPath -eq $resolvedPath) {
                            Write-Error "Removal of directory '$resolvedPath' failed: $exc"
                        }
                        else {
                            Write-Error "Removal of directory '$resolvedPath' failed, because its content could not be (fully) removed: $targetPath`: $exc"
                        }
                    }
                    else {
                        Write-Error $_  # relay error, but in the name of this function
                    }
                    continue
                }
            }
        }
    }
}

Get-ChildItem -Recurse -Directory | `
    Select-Object -ExpandProperty FullName -Property FullName |  Where-Object { 
    (

        ( $_.FullName -match '\\vendor$' -and $_.FullName -notmatch '\\vendor\\' ) -or 

        ( $_.FullName -match '\\vendors$' -and $_.FullName -notmatch '\\vendors\\' ) -or 
        ( $_.FullName -match '\\node_modules$' -and $_.FullName -notmatch '\\node_modules\\')
    )
} | Where-Object {
    # don't match these 
    # wordpress
    $_.FullName -notmatch '\\node_modules\\vendors$' 
} | Where-Object {
    $_.FullName -notmatch 'wp-content'
} | ForEach-Object {
  Write-Host "Removing $($_.FullName)"
  Remove-FileSystemItem -Path $_.FullName -Recurse
 } 

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

The reCAPTCHA verification period has expired. Please reload the page.

You May Also Like…

Squarespace Image Export

To gain continued access to your Squarespace website images after cancelling your subscription you have several...

MySQL 8.x GRANT ALL STATEMENT

-- CREATE CREATE USER 'tgnrestoreuser'@'localhost' IDENTIFIED BY 'AppleSauceLoveBird2024'; GRANT ALL PRIVILEGES ON...

Exetel Opt-Out of CGNAT

If your port forwards and inbound and/or outbound site-to-site VPN's have failed when switching to Exetel due to their...