Uploads fail silently + some other strange behaviour

[AdamVenn]

Thanks for your help with previous issues - I have a new one for you! Thanks in advance for your help.

Steps to reproduce

1. Use Flow Upload to upload a folder containing a folder of many files.
2. Hit 'start', the upload runs. Some files at the beginning of the list fail with 'error' and most read 'succeed'.
3. The errors register in the logs as lock exceptions on the subfolder that was just created to put the files in (race condition?)
4. Retry the failed uploads, which then register success.
5. Check to see if the retried files are there - some are not.
6. Delete the folder, in order to try again.
7. Re-upload the folder, no errors logged, all files indicate success. (Some sort of cache? It already has a record of a successful upload and re-reports it?)
8. Download the folder and compare the files. Some files at the beginning of the list are missing.

Expected behaviour

1. When Flow Upload has to make folders in order to do the upload, it should check that the necessary folders are created and no longer locked before uploading a file into the folder. Maybe it should wait and recheck a few times before declaring failure.
2. Flow Upload should not indicate 'success' if the file is not there.

Actual behaviour

Flow Upload fails to upload when the folder is locked (due to only just being created? (in a different thread?)) and misreports successful uploads.

Server configuration

Operating system: Ubuntu 20.04.1 LTS

Web server: Apache/2.4.46

Database: mysql Ver 15.1 Distrib 10.3.25-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

PHP version: Version: 7.4.3, Memory Limit: 512 MB, Max Execution Time: 3600, Upload max size: 700 GB

Nextcloud version: 19.0.5 (Latest stable at time of writing)

Updated from an older Nextcloud/ownCloud or fresh install: Updated from 19.0.4 today!

Where did you install Nextcloud from: wget https://download.nextcloud.com/server/releases/nextcloud-18.0.4.tar.bz2

List of activated apps:

App list

Enabled: - activity: 2.12.1 - admin_audit: 1.9.0 - bruteforcesettings: 2.0.1 - cloud_federation_api: 1.2.0 - contactsinteraction: 1.0.0 - dav: 1.15.0 - documentserver_community: 0.1.8 - extract: 1.2.5 - federatedfilesharing: 1.9.0 - files: 1.14.0 - files_downloadactivity: 1.8.0 - files_external: 1.10.0 - files_pdfviewer: 1.8.0 - files_rightclick: 0.16.0 - files_sharing: 1.11.0 - files_trashbin: 1.9.0 - files_videoplayer: 1.8.0 - flowupload: 1.1.2 - group_everyone: 0.1.6 - logreader: 2.4.0 - lookup_server_connector: 1.7.0 - metadata: 0.12.0 - oauth2: 1.7.0 - onlyoffice: 6.1.0 - password_policy: 1.9.1 - privacy: 1.3.0 - provisioning_api: 1.9.0 - serverinfo: 1.9.0 - settings: 1.1.0 - sharebymail: 1.9.0 - text: 3.0.1 - theming: 1.10.0 - twofactor_backupcodes: 1.8.0 - updatenotification: 1.9.0 - user_ldap: 1.9.1 - viewer: 1.3.0 - workflowengine: 2.1.0 Disabled: - accessibility - comments - encryption - federation - files_versions - firstrunwizard - nextcloud_announcements - notifications - photos - recommendations - support - survey_client - systemtags

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "19.0.5.2",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "log_type": "syslog",
        "logfile": "",
        "loglevel": 3,
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "trashbin_retention_obligation": "1, 2",
        "ldapUserCleanupInterval": "30",
        "maintenance": false,
        "theme": "",
        "app_install_overwrite": [
            "files_frommail"
        ],
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    },
    "apps": {
        "accessibility": {
            "enabled": "no",
            "installed_version": "1.4.0",
            "types": ""
        },
        "activity": {
            "enabled": "yes",
            "installed_version": "2.12.1",
            "types": "filesystem"
        },
        "admin_audit": {
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": "logging"
        },
        "backgroundjob": {
            "lastjob": "3651"
        },
        "bruteforcesettings": {
            "enabled": "yes",
            "installed_version": "2.0.1",
            "types": ""
        },
        "calendar": {
            "enabled": "no",
            "installed_version": "2.0.3",
            "types": ""
        },
        "cloud_federation_api": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        },
        "comments": {
            "enabled": "no",
            "installed_version": "1.8.0",
            "types": "logging"
        },
        "contactsinteraction": {
            "enabled": "yes",
            "installed_version": "1.0.0",
            "types": "dav"
        },
        "core": {
            "backgroundjobs_mode": "ajax",
            "installed.bundles": "[\"CoreBundle\"]",
            "installedat": "1589901242.7554",
            "lastcron": "1606425363",
            "lastupdateResult": "[]",
            "lastupdatedat": "1606423576",
            "oc.integritycheck.checker": "[]",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "theming.variables": "b494f9fe4e205ef1e2a61df9ff5f797d",
            "updater.secret.created": "1606413618",
            "vendor": "nextcloud"
        },
        "dav": {
            "buildCalendarReminderIndex": "yes",
            "buildCalendarSearchIndex": "yes",
            "chunks_migrated": "1",
            "enabled": "yes",
            "generateBirthdayCalendar": "no",
            "installed_version": "1.15.0",
            "regeneratedBirthdayCalendarsForYearFix": "yes",
            "sendEventReminders": "no",
            "sendInvitations": "no",
            "types": "filesystem"
        },
        "documentserver_community": {
            "enabled": "yes",
            "installed_version": "0.1.8",
            "types": "filesystem"
        },
        "extract": {
            "enabled": "yes",
            "installed_version": "1.2.5",
            "types": ""
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": ""
        },
        "federation": {
            "enabled": "no",
            "installed_version": "1.8.0",
            "types": "authentication"
        },
        "files": {
            "cronjob_scan_files": "500",
            "enabled": "yes",
            "installed_version": "1.14.0",
            "types": "filesystem"
        },
        "files_antivirus": {
            "av_cmd_options": "",
            "av_host": "",
            "av_infected_action": "delete",
            "av_max_file_size": "-1",
            "av_mode": "socket",
            "av_path": "\/usr\/bin\/clamscan",
            "av_port": "0",
            "av_socket": "\/var\/run\/clamav\/clamd.ctl",
            "av_stream_max_length": "26214400",
            "enabled": "no",
            "installed_version": "2.4.1",
            "types": "filesystem,dav"
        },
        "files_downloadactivity": {
            "enabled": "yes",
            "installed_version": "1.8.0",
            "types": "filesystem"
        },
        "files_external": {
            "enabled": "yes",
            "installed_version": "1.10.0",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "1.8.0",
            "types": ""
        },
        "files_rightclick": {
            "enabled": "yes",
            "installed_version": "0.16.0",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "incoming_server2server_share_enabled": "no",
            "installed_version": "1.11.0",
            "lookupServerEnabled": "no",
            "lookupServerUploadEnabled": "no",
            "outgoing_server2server_share_enabled": "yes",
            "types": "filesystem"
        },
        "files_trashbin": {
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": "filesystem,dav"
        },
        "files_versions": {
            "enabled": "no",
            "installed_version": "1.11.0",
            "types": "filesystem,dav"
        },
        "files_videoplayer": {
            "enabled": "yes",
            "installed_version": "1.8.0",
            "types": ""
        },
        "firstrunwizard": {
            "enabled": "no",
            "installed_version": "2.7.0",
            "types": "logging"
        },
        "flowupload": {
            "enabled": "yes",
            "installed_version": "1.1.2",
            "types": ""
        },
        "group_everyone": {
            "enabled": "yes",
            "installed_version": "0.1.6",
            "types": "authentication"
        },
        "logreader": {
            "enabled": "yes",
            "installed_version": "2.4.0",
            "types": ""
        },
        "lookup_server_connector": {
            "enabled": "yes",
            "installed_version": "1.7.0",
            "types": "authentication"
        },
        "metadata": {
            "enabled": "yes",
            "installed_version": "0.12.0",
            "types": ""
        },
        "nextcloud_announcements": {
            "enabled": "no",
            "installed_version": "1.7.0",
            "types": "logging"
        },
        "notifications": {
            "enabled": "no",
            "installed_version": "2.6.0",
            "types": "logging"
        },
        "oauth2": {
            "enabled": "yes",
            "installed_version": "1.7.0",
            "types": "authentication"
        },
        "onlyoffice": {
            "DocumentServerUrl": "***REMOVED SENSITIVE VALUE***/index.php\/apps\/documentserver_community\/",
            "defFormats": "{\"docx\":true,\"pptx\":true,\"xlsx\":true,\"odp\":true,\"ods\":true,\"odt\":true,\"doc\":true,\"ppt\":true,\"xls\":true}",
            "editFormats": "{\"csv\":true,\"docx\":true,\"pptx\":true,\"txt\":true,\"xlsx\":true,\"odp\":true,\"ods\":true,\"odt\":true,\"rtf\":true}",
            "enabled": "yes",
            "installed_version": "6.1.0",
            "sameTab": "true",
            "types": "filesystem"
        },
        "password_policy": {
            "enabled": "yes",
            "enforceHaveIBeenPwned": "1",
            "installed_version": "1.9.1",
            "types": "authentication"
        },
        "photos": {
            "enabled": "no",
            "installed_version": "1.0.0",
            "types": ""
        },
        "privacy": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": ""
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": "prevent_group_restriction"
        },
        "recommendations": {
            "enabled": "no",
            "installed_version": "0.6.0",
            "types": ""
        },
        "serverinfo": {
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": ""
        },
        "settings": {
            "enabled": "yes",
            "installed_version": "1.1.0",
            "types": ""
        },
        "sharebymail": {
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": "filesystem"
        },
        "support": {
            "enabled": "no",
            "installed_version": "1.1.0",
            "types": "session"
        },
        "survey_client": {
            "enabled": "no",
            "installed_version": "1.6.0",
            "types": ""
        },
        "systemtags": {
            "enabled": "no",
            "installed_version": "1.8.0",
            "types": "logging"
        },
        "text": {
            "enabled": "yes",
            "installed_version": "3.0.1",
            "types": "dav"
        },
        "theming": {
            "backgroundMime": "image\/png",
            "cachebuster": "7",
            "color": "#002F7F",
            "enabled": "yes",
            "faviconMime": "image\/png",
            "installed_version": "1.10.0",
            "logoMime": "image\/png",
            "name": "***REMOVED SENSITIVE VALUE***",
            "slogan": "***REMOVED SENSITIVE VALUE***",
            "types": "logging",
            "url": "***REMOVED SENSITIVE VALUE***"
        },
        "twofactor_backupcodes": {
            "enabled": "yes",
            "installed_version": "1.8.0",
            "types": ""
        },
        "updatenotification": {
            "bruteforcesettings": "2.0.1",
            "core": "19.0.2.2",
            "documentserver_community": "0.1.7",
            "enabled": "yes",
            "extract": "1.2.5",
            "installed_version": "1.9.0",
            "metadata": "0.12.0",
            "onlyoffice": "6.0.0",
            "types": "",
            "update_check_errors": "0"
        },
        "user_ldap": {
            "cleanUpJobOffset": "0",
            "enabled": "yes",
            "installed_version": "1.9.1",
            "s01_lastChange": "1606413302",
            "s01has_memberof_filter_support": "1",
            "s01home_folder_naming_rule": "",
            "s01last_jpegPhoto_lookup": "0",
            "s01ldap_agent_password": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_attributes_for_group_search": "",
            "s01ldap_attributes_for_user_search": "",
            "s01ldap_backup_host": "",
            "s01ldap_backup_port": "",
            "s01ldap_base": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_base_groups": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_base_users": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_cache_ttl": "600",
            "s01ldap_configuration_active": "1",
            "s01ldap_default_ppolicy_dn": "",
            "s01ldap_display_name": "displayname",
            "s01ldap_dn": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_dynamic_group_member_url": "",
            "s01ldap_email_attr": "",
            "s01ldap_experienced_admin": "0",
            "s01ldap_expert_username_attr": "",
            "s01ldap_expert_uuid_group_attr": "",
            "s01ldap_expert_uuid_user_attr": "",
            "s01ldap_ext_storage_home_attribute": "",
            "s01ldap_gid_number": "gidNumber",
            "s01ldap_group_display_name": "cn",
            "s01ldap_group_filter": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_group_filter_mode": "0",
            "s01ldap_group_member_assoc_attribute": "member",
            "s01ldap_groupfilter_groups": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_groupfilter_objectclass": "group",
            "s01ldap_host": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_login_filter": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_login_filter_mode": "0",
            "s01ldap_loginfilter_attributes": "",
            "s01ldap_loginfilter_email": "0",
            "s01ldap_loginfilter_username": "1",
            "s01ldap_nested_groups": "0",
            "s01ldap_override_main_server": "",
            "s01ldap_paging_size": "500",
            "s01ldap_port": "389",
            "s01ldap_quota_attr": "",
            "s01ldap_quota_def": "",
            "s01ldap_tls": "0",
            "s01ldap_turn_off_cert_check": "0",
            "s01ldap_turn_on_pwd_change": "0",
            "s01ldap_user_avatar_rule": "default",
            "s01ldap_user_display_name_2": "",
            "s01ldap_user_filter_mode": "0",
            "s01ldap_userfilter_groups": "***REMOVED SENSITIVE VALUE***",
            "s01ldap_userfilter_objectclass": "person",
            "s01ldap_userlist_filter": "***REMOVED SENSITIVE VALUE***",
            "s01use_memberof_to_detect_membership": "1",
            "types": "authentication"
        },
        "viewer": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": ""
        },
        "workflowengine": {
            "enabled": "yes",
            "installed_version": "2.1.0",
            "types": "filesystem"
        }
    }
}

Are you using external storage, if yes which one: local, but have not yet tried uploading to it using Flow Upload

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP, connecting to a Samba 4 AD

LDAP configuration (delete this part if not used)

LDAP config

+-------------------------------+-------------+
| Configuration                 | s01         |
+-------------------------------+-------------+
| hasMemberOfFilterSupport      | 1           |
| homeFolderNamingRule          |             |
| lastJpegPhotoLookup           | 0           |
| ldapAgentName                 | ***         |
| ldapAgentPassword             | ***         |
| ldapAttributesForGroupSearch  |             |
| ldapAttributesForUserSearch   |             |
| ldapBackupHost                |             |
| ldapBackupPort                |             |
| ldapBase                      | ***         |
| ldapBaseGroups                | ***         |
| ldapBaseUsers                 | ***         |
| ldapCacheTTL                  | 600         |
| ldapConfigurationActive       | 1           |
| ldapDefaultPPolicyDN          |             |
| ldapDynamicGroupMemberURL     |             |
| ldapEmailAttribute            |             |
| ldapExperiencedAdmin          | 0           |
| ldapExpertUUIDGroupAttr       |             |
| ldapExpertUUIDUserAttr        |             |
| ldapExpertUsernameAttr        |             |
| ldapExtStorageHomeAttribute   |             |
| ldapGidNumber                 | gidNumber   |
| ldapGroupDisplayName          | cn          |
| ldapGroupFilter               | ***         |
| ldapGroupFilterGroups         | ***         |
| ldapGroupFilterMode           | 0           |
| ldapGroupFilterObjectclass    | group       |
| ldapGroupMemberAssocAttr      | member      |
| ldapHost                      | ***         |
| ldapIgnoreNamingRules         |             |
| ldapLoginFilter               | ***         |
| ldapLoginFilterAttributes     |             |
| ldapLoginFilterEmail          | 0           |
| ldapLoginFilterMode           | 0           |
| ldapLoginFilterUsername       | 1           |
| ldapMatchingRuleInChainState  | unknown     |
| ldapNestedGroups              | 0           |
| ldapOverrideMainServer        |             |
| ldapPagingSize                | 500         |
| ldapPort                      | 389         |
| ldapQuotaAttribute            |             |
| ldapQuotaDefault              |             |
| ldapTLS                       | 0           |
| ldapUserAvatarRule            | default     |
| ldapUserDisplayName           | displayname |
| ldapUserDisplayName2          |             |
| ldapUserFilter                | ***         |
| ldapUserFilterGroups          | ***         |
| ldapUserFilterMode            | 0           |
| ldapUserFilterObjectclass     | person      |
| ldapUuidGroupAttribute        | auto        |
| ldapUuidUserAttribute         | auto        |
| turnOffCertCheck              | 0           |
| turnOnPasswordChange          | 0           |
| useMemberOfToDetectMembership | 1           |
+-------------------------------+-------------+

Client configuration

Browser: Firefox 83

Operating system: Pop OS

Logs

Web server error log

Web server error log

Available on request :)

Nextcloud log (data/nextcloud.log)

Nextcloud log

Available on request :)

Browser log

Browser log

Available on request :)

[JonathanTreffler]

2. Flow Upload should not indicate 'success' if the file is not there.

I am quite sure that cannot happen. The problem must be somewhere else.
The server only sends 200 codes if the files sucessfully got uploaded and assembled.

• When Flow Upload has to make folders in order to do the upload, it should check that the necessary folders are created and no longer locked before uploading a file into the folder. Maybe it should wait and recheck a few times before declaring failure.

I honestly don't quite understand what you mean by that. Folders are not getting locked !?
The way it works:

1. Upload all chunks of the file
2. Check if folder exists
3. Create if it doesn't
4. Assemble File with file locking

And i don't see a scenario where that fails.
Theoretically the folder could get created by two processes at exactly the same time, but the time from step 2 and 3 ist tiny.
We are using the nextcloud wrappers for checking if the folder exists and creating it, so these might have protections against race conditions.

We could add a mechanism to retry the folder creation if mkdir fails, but i honestly don't think that is the cause of this problem.

Are you using Nextcloud External Storages ?
Are you using a reverse proxy ?

What i think is actually hapening:

The Server is either not very powerful or the files are very big
The chunks get uploaded
The final chunk is getting uploaded and within the same request the chunks get assembled.
The assembling takes too long wich causes a timeout (maybe just on a reverse proxy)
The frontend recieves a 408 or 504 http error and displays error, because the http code was not 200.
The Server is still busy assembling the chunks.
You click retry while the server is still assembling from the first try.
The frontend checks wich chunks are already on the server -> every chunk is there so the chunk assembling gets triggered again.
Two processes of flowupload try to copy the chunks into the same file / try to lock the file at the same time.
One or both of the processes fail.

[JonathanTreffler]

The probability of a race condition happening with multiple files within the same upload, with only 4 concurrent uploads and a automatic second try if any chunk or the assembly fails, is incredibly tiny / nonexistent.

[JonathanTreffler]

Also if the mkdir command fails it would never send a 200 code, so it could not be shown as sucessful in the frontend

[e-alfred]

@AdamVenn Do you use external shares (SMB, FTP, etc) where you want to upload files to using Flowupload? Some types of them still are problematic.

[AdamVenn]

Thanks for the responses @JonathanTreffler and @e-alfred. Sorry for the delay in replying.
The Nextcloud is in a VM, so it could be that it's not powerful enough and needs dedicated hardware.
We are not using a reverse proxy and the problem uploads have been to internal storage. The files are not normally very large, generally staying under 50MB each, though there can be many of them.
Your description of a potential failure with large files/slow server assembling is very useful - I guess I need to try the same kind of upload on a Nextcloud with faster disk IO and processor.
It sounds like Flow Upload relies on the proper functioning of the file handling underneath. I have had multiple occasions now where uploads have been missing files, and it is concerning that no errors are being logged in these cases. It would seem the Nextcloud file system is not providing your application with correct responses.
If there is no information I can provide you to verify that this is the case, and there are no further options apart from trying a different machine, then I suppose you might as well close the issue.