I haven't yet determined a good number of threads to use, we'll see.
Also have to change how the directories are being handled, kind of a
waste to push every directory to the queue, would be faster if the
batches were bigger.
I also still have to work on using fstat across a tree, this will bring
super speed ups. Output is a bit different, parsing is more complex and
how we handle things will be a bit different.
Made the threaded console to batch messages and so I could manually
flush or clear them. At some point I would consider a safety maximum
buffer size that triggers it to auto flush. This worked out really well,
though I have to see why in some cases lines appear to double up still,
could be something with the process not completing when I expect it to.
This is possible a naive thread implementation, since it pushes a
directory for every thread which seems too drastic. I'd like to see how
much better it works without all the context switches. It's also a
matter of figuring out how much to handle yourself before letting
another thread join in. Right now the threads don't branch out too much
since I think they basically do a breadth-first search, though I have to
double check on that.
Still to come, trying to safely work with fstat across multiple
directories. It's fast, but on the console the script would appear to
stall as it parses everything, so I'd still want to break it down
somewhat so you can see the script making visible progress. I would
also prefer this because then console messages wouldn't be so short and
blocky.
Improvements to come!
I was trying to use `p4 have` for speed, but it doesn't seem to work
with files that are added to a changelist but not to a repo. So I had
to resort back to `p4 fstat`.
So the speed of the script is much faster than before, though it
actually still has much room for improvement, it will just be more
complicated. Calling 'p4 fstat' on the entire directory will give you
everything you need up front, it's just they're in depot paths, which
makes thing a little annoying to parse when you have workspace mappings
that move things around so the local path may differ from the depot
path, and it becomes harder to determine 100% that you're referring to
the same file. And I don't want to have to call p4 on every file to be
sure of that, what I'm doing now is the easiest safest way to be sure of
that, as far as I know.
Another way to speed this up is to add thread crawlers, I'm just not yet
sure with HDDs and SSDs how many threads is a good idea to use.
Brian