Commit Graph

12 Commits

Author SHA1 Message Date
Brian b3b960e9ef Fixed console to exit properly, wasn't finished frankly.
Script now exits as expected.
2014-05-09 14:16:18 -06:00
Brian b3051f8dc8 Fixed mixing of unix/windows paths. Need to test this works cross platform.
Also removed PressEnter. Added global basename function so we can
override which version we're using, right now I'm seeing if
ntpath.basename works for all cases.
2014-05-09 12:15:32 -06:00
Brian 8bb78e7c02 Added exception catching with printing to thread-safe console. Removed press enter call.
Changed output a little bit.

Also just realized it actually should be easy to parse `p4 fstat ...`, I
just need to crab the clientFile output, and this script should be sped
up substantially. I need to figure out the best way to break this down,
don't want it to be called on a huge directory, but each subdirectory to
split up the work.  That said, that would miss the top level files.  A
good alternative to not waiting is to see if I can grab the process
output while it's working, instead of waiting for it to be done.  This
would actually work perfectly; it's just tricky trying to figure out if
I can break this up. This would also still delay the start of the
script.  Could do a mix of local and tree based fstat.  Start with local
and switch to the tree.
2014-05-09 11:51:59 -06:00
Brian d3fdef1342 Removed thread shutdown print, and bumped up thread count.
I haven't yet determined a good number of threads to use, we'll see.

Also have to change how the directories are being handled, kind of a
waste to push every directory to the queue, would be faster if the
batches were bigger.

I also still have to work on using fstat across a tree, this will bring
super speed ups.  Output is a bit different, parsing is more complex and
how we handle things will be a bit different.
2014-05-09 11:25:53 -06:00
Brian 3eb7a78339 Fixed a serious output bug, buffers weren't being cleared after flushed.
So I obviously feel much better about this version, looks like it works
how it should.
2014-05-09 11:12:41 -06:00
Brian 97da25ce38 Threaded console, threaded cleanup. Yes!
Made the threaded console to batch messages and so I could manually
flush or clear them.  At some point I would consider a safety maximum
buffer size that triggers it to auto flush. This worked out really well,
though I have to see why in some cases lines appear to double up still,
could be something with the process not completing when I expect it to.

This is possible a naive thread implementation, since it pushes a
directory for every thread which seems too drastic. I'd like to see how
much better it works without all the context switches.  It's also a
matter of figuring out how much to handle yourself before letting
another thread join in.  Right now the threads don't branch out too much
since I think they basically do a breadth-first search, though I have to
double check on that.

Still to come, trying to safely work with fstat across multiple
directories. It's fast, but on the console the script would appear to
stall as it parses everything, so I'd still want to break it down
somewhat so you can see the script making visible progress.  I would
also prefer this because then console messages wouldn't be so short and
blocky.

Improvements to come!
2014-05-08 22:55:12 -06:00
Brian 2b14c4a273 Working on threaded support. 2014-05-08 21:05:55 -06:00
Brian 1f4b52e3a9 Added the version of Python I've tested with.
I'm working on making sure it works in Python 3.4.0
2014-05-08 19:27:29 -06:00
Brian b327058ccb Added a fix for old now unversioned files with readonly set. 2014-05-08 19:17:36 -06:00
Brian 2bb0fa671d Fixed a bug from old code so the script would work, tweaked output.
I was trying to use `p4 have` for speed, but it doesn't seem to work
with files that are added to a changelist but not to a repo.  So I had
to resort back to `p4 fstat`.
2014-05-08 19:05:07 -06:00
Brian 266c5555ba Improved crawling speed, also cleaned up the output.
So the speed of the script is much faster than before, though it
actually still has much room for improvement, it will just be more
complicated.  Calling 'p4 fstat' on the entire directory will give you
everything you need up front, it's just they're in depot paths, which
makes thing a little annoying to parse when you have workspace mappings
that move things around so the local path may differ from the depot
path, and it becomes harder to determine 100% that you're referring to
the same file.  And I don't want to have to call p4 on every file to be
sure of that, what I'm doing now is the easiest safest way to be sure of
that, as far as I know.

Another way to speed this up is to add thread crawlers, I'm just not yet
sure with HDDs and SSDs how many threads is a good idea to use.
2014-05-08 19:05:07 -06:00
Brian 27e2e32f7e Added the basic script. 2014-05-08 15:37:57 -06:00