From d7ff7a6646459d2323b0a16163c6f9aa0f31804d Mon Sep 17 00:00:00 2001 From: Brian Date: Fri, 9 May 2014 15:24:12 -0600 Subject: [PATCH 1/4] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index abd326d..ce65e1e 100644 --- a/README.md +++ b/README.md @@ -6,3 +6,7 @@ Removes unversioned files from perforce repository. Script is in beta, though it This script does parse __.p4ignore__ ignore files, compiles the fields as regex, and scans every directory and file against the local and parent __.p4ignore__ files. This is my first time doing something like this, and I just realized this isn't actually correct; I need to update how things are ignored to follow the [spec](http://www.perforce.com/perforce/r12.1/manuals/cmdref/env.P4IGNORE.html), since it's not straight up regex. **Files are currently permanently deleted, so use this at your own risk.** + +Raw Tests +=================== +On a 133GB Directory with 15,700 Folders and 153,415 Files, the script ran for 11m and 16.35s. This will only get better. From bcff97c22ced27c377801eed25b595277f4578bc Mon Sep 17 00:00:00 2001 From: Brian Date: Fri, 9 May 2014 15:40:13 -0600 Subject: [PATCH 2/4] Update README.md --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ce65e1e..7c7c053 100644 --- a/README.md +++ b/README.md @@ -9,4 +9,6 @@ This script does parse __.p4ignore__ ignore files, compiles the fields as regex, Raw Tests =================== -On a 133GB Directory with 15,700 Folders and 153,415 Files, the script ran for 11m and 16.35s. This will only get better. +With Python 2.7.6 on a 133GB Directory with 15,700 Folders and 153,415 Files, the script ran for 11m and 16.35s. This will only get better. + +With Python 3.4.0 on the same directory, the script ran for 3m and 00.86s. At this point the directory was already stripped of unversioned files, but that's not the longest part of the process, so this version of Python is just that much faster. I'll have more concrete tests later when things are changing less in code. From beab6428c11ddac6b5b84a3efa979ce0a6ac5816 Mon Sep 17 00:00:00 2001 From: Brian Date: Fri, 9 May 2014 17:19:09 -0600 Subject: [PATCH 3/4] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 7c7c053..02b16e7 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,12 @@ This script does parse __.p4ignore__ ignore files, compiles the fields as regex, **Files are currently permanently deleted, so use this at your own risk.** +**This is currently Windows only, but it will be updated to work on Linux** + Raw Tests =================== With Python 2.7.6 on a 133GB Directory with 15,700 Folders and 153,415 Files, the script ran for 11m and 16.35s. This will only get better. With Python 3.4.0 on the same directory, the script ran for 3m and 00.86s. At this point the directory was already stripped of unversioned files, but that's not the longest part of the process, so this version of Python is just that much faster. I'll have more concrete tests later when things are changing less in code. + +Notes From 3d110652994bb573947fb935a0b91cd2438523bf Mon Sep 17 00:00:00 2001 From: Brian Date: Tue, 13 May 2014 16:56:13 -0600 Subject: [PATCH 4/4] Update README.md --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 02b16e7..49e0903 100644 --- a/README.md +++ b/README.md @@ -15,4 +15,10 @@ With Python 2.7.6 on a 133GB Directory with 15,700 Folders and 153,415 Files, th With Python 3.4.0 on the same directory, the script ran for 3m and 00.86s. At this point the directory was already stripped of unversioned files, but that's not the longest part of the process, so this version of Python is just that much faster. I'll have more concrete tests later when things are changing less in code. +**UPDATE** +To give you an idea of the improvement, running the new script in the testing branch with Python 3.4.0 has a runtime of 3m and 44.44s, singlethreaded. The previous test was done with 100 threads. So, once I make this multi-threaded, it will be blazing. + + Notes +=================== +Besides making the new script multi-threaded, I'm looking at a way of improving the os.walk speed, I see there's a betterwalk module I can take advantage of. I noticed when iterating directories for deletion it's super fast, but iterating files makes it super slow for some reason; so, I want to look into this.