v0.9.9.3 is still in the dev/test phase

While testing the new Scanning Engine this evening, I found an issue that will affect a certain group of users. The users that will be affected by the issue I found (including myself...), has to do with the Asset Store-5.x directory being setup as a junction to another location on the system. Most people do this, so that they don't run out of space on their C: drive, due to all of the packages that they've purchased/downloaded.

Because accessing the file, without .Net being involved, because the file itself contains a ReparsePoint attribute, the OS instinctively decides to add the file to the System Page Files Standby (Reserved) memory, which overtime, can become quite large in size.

Because the process is scanning virtually every package file, this causes the system to eventually run out of memory. The problem is that unity only supports .Net v3.5 (CLR 2.0), but MemoryMappedFile management wasn't added until .Net 4.x, which isn't currently supported by Unity at the moment. I've researched the issue and have found one backport to .net 2.0, but it relies on Local Operating system files from Windows.

I'm currently modifying the process as follows:

1) Create a new Assembly, which will be set to Windows only, in the Unity Editor. 
  • Because the system is in compiled DLL form, I ca't utilize Unity's Pre-Compiler options, for C#
2) Determine if the unity package file is being accessed via a Reparse Point Attribute check (Done)
  • If not, no need to go any further...
3) Detect if the OS is running under Windows
  • If not, no need to go any further (exception of checking for similar Mac/Linux issues as well)
4) Dynamically load up the additional .Net Windows Only assembly, and use the Win32API calls for MemeoryMappedFiles, and Flush the View from Memory, and got on to the next file...

I'm hoping to have this re-coded and testing wrapped up by February 15th. i know I missed the Jan 15th deadline, but a lot of changes have been made to the scanning engine, and a lot of code has to be re-factored to utilize the updated functions.

I've also decided that the UPAI, from now on, will run in the background, in a separate thread, with below normal thread priority, so as to not interfere with your use of Unity for Game Development. I'm thinking of leaving in the dedicated scan, as an option, if the user wants to just let it run, by itself.

Also, i may have to limit the size of the unity packages that can be indexed, due to processing time required. One of the issues I ran into was with the Unity Technologies Adam Project files over 2.0 GB in size. Currently the system is handling well with files sizes up to 1.5 GB. 

also, the scan times have been greatly improved, and the process is running as multi-threaded, which means multiple Packages/Assets are being scanned simultaneously. This next version will also have the first iteration of parsing all currently Supported 3D Model formats, currently supported by Unity, out of the box. this doesn't include Cinem4D, 3DS Max, nor Maya files, currently, and aren't planned for any future release, as of yet.