Problem with folders containing many files

Home Forums PFM Developer Support Problem with folders containing many files

This topic contains 5 posts, has 2 voices, and was last updated by  Joe Lowe 2 years, 6 months.

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts

  • Dhebug
    Posts: 12

    Hi,
    one of my users reported that he could not find some of the files he had saved.

    I did some debugging and validated that the missing files were correctly added to the PfmMarshallerListOp list of files, but they were indeed not visible in either a Windows explorer window or when calling DIR from the command line.

    I then exported the content of the directory listing from both my internal data and from DIR. Internally I reported 8590 files, while DIR only shown 8582.

    These files are automatically generated by a third party middleware, and are using some fancy generated names with some common prefixes and extensions, so I was wondering if that could not be some obscure hashing bug. I checked on my side, looked at what pfm trace returned, but could not find anything obvious.

    Here is an example of some of the file names:

    4666.gts
    4666.uasset
    4666_13488d3d.uasset
    4666_1a18e371.uasset
    (…)
    4666_3cbbee4.uasset
    4666_435fb7f4.uasset
    4666_44fa02be.uasset
    4666_4666_36fbe3e2.gtp
    4666_4666_435fb7f4.gtp
    4666_4666_561ee765.gtp
    4666_4666_73ebbde1.gtp
    (…)
    4666_4666_e915b767.gtp
    4666_4a2fe75b.uasset
    4666_509f2661.uasset
    4666_561ee765.uasset
    (…)
    4666_f7b3a085.uasset
    4666_f7d3b23.uasset
    4666_f9d8513a.uasset
    4666_fa1865d3.uasset
    (…)
    4666_mips.gtp

    in this particular case, the missing one is 4666.gts which is nowhere to be seen.

    Suggestions welcome :)
    Thanks.


    Joe Lowe
    Posts: 101

    Try to reproduce using a sample zip or iso containing the same folder layout and zero length files of the same names, and e-mail me the sample. If there is something here for me to fix, I need to see it quick to get in build 187.


    Dhebug
    Posts: 12

    I tried to reproduce with the zip file idea, mounting it, and diffing with my source folder, the issue did not happen.

    I’ve done a bit more test, using the command shell, and I got that:
    ——————————————————
    C:\Volumes\Project\Content\GraniteSDK\Baking>dir 0066_acf9d70d.uasset

    01.06.2017 20:36 1 253 0066_acf9d70d.uasset
    ——————————————————
    C:\Volumes\Project\Content\GraniteSDK\Baking>dir 0066_acf9d70d*.*

    File Not Found
    ——————————————————
    C:\Volumes\Project\Content\GraniteSDK\Baking>dir 0066_acf9d70d.*

    File Not Found
    ——————————————————
    C:\Volumes\Project\Content\GraniteSDK\Baking>dir 0066_acf9d70d.uasset

    01.06.2017 20:36 1 253 0066_acf9d70d.uasset
    ——————————————————
    C:\Volumes\Project\Content\GraniteSDK\Baking>dir 0066_acf9d70d.uasse?

    File Not Found
    ——————————————————
    C:\Volumes\Project\Content\GraniteSDK\Baking>attrib 0066_acf9d70d.uasset
    I C:\Volumes\Project\Content\GraniteSDK\Baking\0066_acf9d70d.uasset
    ——————————————————
    C:\Volumes\Project\Content\GraniteSDK\Baking>copy 0066_acf9d70d.uasset ..
    1 file(s) copied.
    ——————————————————

    So basically, when trying to access the file by its full name, everything works (dir, attrib, copy, … find it just fine).

    When using wildcards, or generally speaking “directory listing”, then it’s not found.

    That could be a bug in my implementation of PfmMarshallerListOp and PfmMarshallerListEndOp, but I’m not sure how I can debug that.


    Dhebug
    Posts: 12

    Hmm, I may have found the issue, I possibly misunderstood the sample code, more exactly the meaning of the value returned by “Add” and “Add8” in the PfmMarshallerListOp operation.

    My function looks like that:

    int EntityList::List(PfmMarshallerListOp* op, uint8_t& noMore)
    {
      noMore = 0;
      if (Entity* folder = FindOpenFile(op->OpenId()))
      {
        std::pair<uint64, uint64> folderListId(folder->GetInstanceID(), op->ListId());
        auto findIt = m_FileLists.find(folderListId);
        if (findIt == m_FileLists.end())
        {
          // New list, need to get the list of child
          Entity* entity = GetEntityIfExist(folder->GetInstanceID());
          dwAssertFmt(entity, "Entity not found, we have a big problem there.");
          m_FileLists[folderListId] = entity->GetChildList();
        }
    
        std::vector<uint64>& childList(m_FileLists[folderListId]);
        while (!childList.empty())
        {
          uint64 childId = childList.back();
          childList.pop_back();
    
          Entity* childEntity = GetEntityIfExist(childId); // This one may or may not be valid
    
          PfmAttribs attribs;
          childEntity->GetAttributes(attribs);
          if (!op->Add8(&attribs, entityData->GetName().c_str()))
          {
            return pfmErrorSuccess; // Still some to go
          }
        }
        noMore = 1;
        return pfmErrorSuccess;
      }
      return pfmErrorNotFound;
    }

    The files that are missing are actually hitting the line with “// still some to go”, meaning that Add8 returned false.

    Does that mean that I should change my code so I when I get false, I re-register the file on which things failed on the next call?


    Joe Lowe
    Posts: 101

    If the list->Add() call returns false, there was not enough room in the output buffer for the specified file, and it is not in the list that will be returned to the client.

Viewing 5 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.