p4RemoveUnversioned

Removes unversioned files from perforce repository. Script is in beta, though it works. It's a little slow due to the way fstat is used, but that will be changed soon, so the speed up should be enormous once that's done, up to 100x or more.

This script does parse .p4ignore ignore files, compiles the fields as regex, and scans every directory and file against the local and parent .p4ignore files. This is my first time doing something like this, and I just realized this isn't actually correct; I need to update how things are ignored to follow the spec, since it's not straight up regex.

Files are currently permanently deleted, so use this at your own risk.

This is currently Windows only, but it will be updated to work on Linux

Raw Tests

With Python 2.7.6 on a 133GB Directory with 15,700 Folders and 153,415 Files, the script ran for 11m and 16.35s. This will only get better.

With Python 3.4.0 on the same directory, the script ran for 3m and 00.86s. At this point the directory was already stripped of unversioned files, but that's not the longest part of the process, so this version of Python is just that much faster. I'll have more concrete tests later when things are changing less in code.

UPDATE To give you an idea of the improvement, running the new script in the testing branch with Python 3.4.0 has a runtime of 3m and 44.44s, singlethreaded. The previous test was done with 100 threads. So, once I make this multi-threaded, it will be blazing.

Notes

Besides making the new script multi-threaded, I'm looking at a way of improving the os.walk speed, I see there's a betterwalk module I can take advantage of. I noticed when iterating directories for deletion it's super fast, but iterating files makes it super slow for some reason; so, I want to look into this.